OpenSearch
OpenSearch is the core data storage component of the Observability Logging stack. It replaced ElasticSearch and Kibana in 2021.
Building
Easiest path to building locally is to use a container on a Debian Bullseye WMCS instance >= g3.cores2.ram4.disk20
The commands below are specific to version 2.0.0. Subsequent versions may require these steps to be updated.
Environment
# Install podman and rootlesskit
sudo apt update && sudo apt install -y podman rootlesskit
# Add user subuid/gid settings
echo $(whoami):100000:65536 | sudo tee -a /etc/subuid
echo $(whoami):100000:65536 | sudo tee -a /etc/subgid
# Add registry
echo 'unqualified-search-registries = ["docker.io"]' | sudo tee -a /etc/containers/registries.conf
# Reboot (https://github.com/containers/podman/issues/3890)
sudo reboot
# Once back up and logged in, enable unprivileged_userns_clone
sudo sysctl kernel.unprivileged_userns_clone=1
OpenSearch
# Clone repo
git clone https://github.com/opensearch-project/OpenSearch.git
# Enter OpenSearch repo
cd ~/OpenSearch
# Checkout 2.0.0
git checkout tags/2.0.0
# Give podman permissions on source dir
podman unshare chown -R 1000:1000 .
# Enter the container
podman run -it --rm --user gradle --net host --workdir /home/gradle/opensearch -v $(pwd):/home/gradle/opensearch:Z gradle:jdk11 /bin/bash
# Build
./gradlew -Dbuild.snapshot=false distribution:packages:deb:build
# Exit the container
exit
# Move the compiled deb package to home dir
sudo mv distribution/packages/deb/build/distributions/opensearch*.deb ~/
# Optionally change ownership to yourself
sudo chown $(id -u):$(id -g) ~/opensearch*.deb
OpenSearch Dashboards
Building OpenSearch Dashboards is disk intensive. Greater than 10GB free disk space is recommended.
# Clone repo
git clone https://github.com/opensearch-project/OpenSearch-Dashboards.git
cd ~/OpenSearch-Dashboards
# Checkout version 2.0.0
git checkout tags/2.0.0
# Give podman permissions on source dir
podman unshare chown -R 1000:1000 .
# Enter the container
podman run -it --rm --user root --net host -v $(pwd):/home/node/osd:Z node:$(cat .node-version)-buster /bin/bash
# Install dependencies
apt update && apt install -y ruby ruby-dev rubygems build-essential
gem install --no-ri --no-rdoc fpm
# Switch to unprivileged node user
su - node
# Change to build dir
cd /home/node/osd
# Bootstrap
yarn osd bootstrap
# Build
node scripts/build --deb --all-platforms --release
# Exit the container
exit
# Move the compiled deb package to home dir
sudo mv target/opensearch-dashboards*.deb ~/
# Optionally change ownership to yourself
sudo chown $(id -u):$(id -g) ~/opensearch-dashboards*.deb
Operations
Cookbooks for OpenSearch can be run from cumin hosts.
Rolling restarting OpenSearch daemons in a cluster
sudo cookbook sre.opensearch.roll-restart-reboot --alias datahubsearch --reason 'operation rationale' restart_daemons
Rolling reboot OpenSearch hosts in a cluster
sudo cookbook sre.opensearch.roll-restart-reboot --alias datahubsearch --reason 'operation rationale' reboot
Troubleshooting
- On a newly provisioned node:
Failed to create directory or subvolume "/run/opensearch-(...)": Permission denied- Run
sudo systemd-tmpfiles --create
- Run
No such file or directory - A directory component in /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py(...).lock does not exist or is a dangling symbolic link- Run
sudo mkdir -p /usr/lib/nagios/plugins
- Run