Jump to content

OpenSearch

From Wikitech

OpenSearch is the core data storage component of the Observability Logging stack. It replaced ElasticSearch and Kibana in 2021.

Building

Easiest path to building locally is to use a container on a Debian Bullseye WMCS instance >= g3.cores2.ram4.disk20

The commands below are specific to version 2.0.0. Subsequent versions may require these steps to be updated.

Environment

# Install podman and rootlesskit
sudo apt update && sudo apt install -y podman rootlesskit
# Add user subuid/gid settings
echo $(whoami):100000:65536 | sudo tee -a /etc/subuid
echo $(whoami):100000:65536 | sudo tee -a /etc/subgid
# Add registry
echo 'unqualified-search-registries = ["docker.io"]' | sudo tee -a /etc/containers/registries.conf
# Reboot (https://github.com/containers/podman/issues/3890)
sudo reboot
# Once back up and logged in, enable unprivileged_userns_clone
sudo sysctl kernel.unprivileged_userns_clone=1

OpenSearch

# Clone repo
git clone https://github.com/opensearch-project/OpenSearch.git
# Enter OpenSearch repo
cd ~/OpenSearch
# Checkout 2.0.0
git checkout tags/2.0.0
# Give podman permissions on source dir
podman unshare chown -R 1000:1000 .
# Enter the container
podman run -it --rm --user gradle --net host --workdir /home/gradle/opensearch -v $(pwd):/home/gradle/opensearch:Z gradle:jdk11 /bin/bash

#  Build
./gradlew -Dbuild.snapshot=false distribution:packages:deb:build
#  Exit the container
exit

# Move the compiled deb package to home dir
sudo mv distribution/packages/deb/build/distributions/opensearch*.deb ~/
# Optionally change ownership to yourself
sudo chown $(id -u):$(id -g) ~/opensearch*.deb

OpenSearch Dashboards

Building OpenSearch Dashboards is disk intensive. Greater than 10GB free disk space is recommended.
# Clone repo
git clone https://github.com/opensearch-project/OpenSearch-Dashboards.git
cd ~/OpenSearch-Dashboards
# Checkout version 2.0.0
git checkout tags/2.0.0
# Give podman permissions on source dir
podman unshare chown -R 1000:1000 .
# Enter the container
podman run -it --rm --user root --net host -v $(pwd):/home/node/osd:Z node:$(cat .node-version)-buster /bin/bash

#   Install dependencies
apt update && apt install -y ruby ruby-dev rubygems build-essential
gem install --no-ri --no-rdoc fpm
#   Switch to unprivileged node user
su - node
#   Change to build dir
cd /home/node/osd
#   Bootstrap
yarn osd bootstrap
#   Build
node scripts/build --deb --all-platforms --release
#   Exit the container
exit

# Move the compiled deb package to home dir
sudo mv target/opensearch-dashboards*.deb ~/
# Optionally change ownership to yourself
sudo chown $(id -u):$(id -g) ~/opensearch-dashboards*.deb

Operations

Cookbooks for OpenSearch can be run from cumin hosts.

Rolling restarting OpenSearch daemons in a cluster

sudo cookbook sre.opensearch.roll-restart-reboot --alias datahubsearch --reason 'operation rationale' restart_daemons

Rolling reboot OpenSearch hosts in a cluster

sudo cookbook sre.opensearch.roll-restart-reboot --alias datahubsearch --reason 'operation rationale' reboot

Troubleshooting

  • On a newly provisioned node:
    • Failed to create directory or subvolume "/run/opensearch-(...)": Permission denied
      • Run sudo systemd-tmpfiles --create
    • No such file or directory - A directory component in /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py(...).lock does not exist or is a dangling symbolic link
      • Run sudo mkdir -p /usr/lib/nagios/plugins