OpenSearch

OpenSearch is the core data storage component of the Observability Logging stack. It replaced ElasticSearch and Kibana in 2021.

Building

Easiest path to building locally is to use a container on a Debian Bullseye WMCS instance >= g3.cores2.ram4.disk20

The commands below are specific to version 2.0.0. Subsequent versions may require these steps to be updated.

Environment

# Install podman and rootlesskit
sudo apt update && sudo apt install -y podman rootlesskit
# Add user subuid/gid settings
echo $(whoami):100000:65536 | sudo tee -a /etc/subuid
echo $(whoami):100000:65536 | sudo tee -a /etc/subgid
# Add registry
echo 'unqualified-search-registries = ["docker.io"]' | sudo tee -a /etc/containers/registries.conf
# Reboot (https://github.com/containers/podman/issues/3890)
sudo reboot
# Once back up and logged in, enable unprivileged_userns_clone
sudo sysctl kernel.unprivileged_userns_clone=1

OpenSearch

# Clone repo
git clone https://github.com/opensearch-project/OpenSearch.git
# Enter OpenSearch repo
cd ~/OpenSearch
# Checkout 2.0.0
git checkout tags/2.0.0
# Give podman permissions on source dir
podman unshare chown -R 1000:1000 .
# Enter the container
podman run -it --rm --user gradle --net host --workdir /home/gradle/opensearch -v $(pwd):/home/gradle/opensearch:Z gradle:jdk11 /bin/bash

#  Build
./gradlew -Dbuild.snapshot=false distribution:packages:deb:build
#  Exit the container
exit

# Move the compiled deb package to home dir
sudo mv distribution/packages/deb/build/distributions/opensearch*.deb ~/
# Optionally change ownership to yourself
sudo chown $(id -u):$(id -g) ~/opensearch*.deb

OpenSearch Dashboards

Building OpenSearch Dashboards is disk intensive. Greater than 10GB free disk space is recommended.

# Clone repo
git clone https://github.com/opensearch-project/OpenSearch-Dashboards.git
cd ~/OpenSearch-Dashboards
# Checkout version 2.0.0
git checkout tags/2.0.0
# Give podman permissions on source dir
podman unshare chown -R 1000:1000 .
# Enter the container
podman run -it --rm --user root --net host -v $(pwd):/home/node/osd:Z node:$(cat .node-version)-buster /bin/bash

#   Install dependencies
apt update && apt install -y ruby ruby-dev rubygems build-essential
gem install --no-ri --no-rdoc fpm
#   Switch to unprivileged node user
su - node
#   Change to build dir
cd /home/node/osd
#   Bootstrap
yarn osd bootstrap
#   Build
node scripts/build --deb --all-platforms --release
#   Exit the container
exit

# Move the compiled deb package to home dir
sudo mv target/opensearch-dashboards*.deb ~/
# Optionally change ownership to yourself
sudo chown $(id -u):$(id -g) ~/opensearch-dashboards*.deb

Operations

Cookbooks for OpenSearch can be run from cumin hosts.

Rolling restarting OpenSearch daemons in a cluster

sudo cookbook sre.opensearch.roll-restart-reboot --alias datahubsearch --reason 'operation rationale' restart_daemons

Rolling reboot OpenSearch hosts in a cluster

sudo cookbook sre.opensearch.roll-restart-reboot --alias datahubsearch --reason 'operation rationale' reboot

Troubleshooting

• On a newly provisioned node:
  • Failed to create directory or subvolume "/run/opensearch-(...)": Permission denied
    • Run sudo systemd-tmpfiles --create
  • No such file or directory - A directory component in /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py(...).lock does not exist or is a dangling symbolic link
    • Run sudo mkdir -p /usr/lib/nagios/plugins