Data Platform/Systems/OpenSearch-on-K8s/Administration
OpenSearch on K8s
Intended audience: SREs who operate the OpenSearch on K8s platform.
If you are a service owner, or are interested in deploying on the platform, you may be more interested in this page.
Dashboards and Alerts
Dashboards
See our OpenSearch on K8s dashboard. The dashboard contains a number of metrics we use to gauge health.
Look out for the 'interval' dropdown
Unlike other WMF dashboard, OpenSearch on K8s contains an "interval" dropdown, used to get a rolling average. If the panels are showing up blank, try to lengthen the interval. But you should also realize if you make the interval too long, it might confuse things as well.
For example, we had an incident where latency spiked to about 10x its normal value. After the incident was resolved, we still had the interval set to 24h. Thus, it was still including periods of extremely high latency in its graphs, making it seem like the incident was still ongoing. We are working to address this confusion in in T417230 .
Important Dashboard Panels
- The "network probes" panel, showing the latency of all OpenSearch clusters per DC.
- Disk Usage - Nothing good happens when you run out of disk space!
- Memory usage - amount of memory used per pod
- Search latency
You can find alerts in the Wikimedia Foundation's alert repo .
Alerting
These alerts will notify DPE SRE once your cluster is in "production" state in service.yaml in Puppet.
As stated in the "Service Availability/Expectations" section, DPE SRE will respond to the alerts within business hours.
The table below describes whether or not the application owner has permissions to fix the problem. This is not meant to imply the application owner must fix the problem. Generally speaking, the platform owner (DPE SRE) is the right team to deal with alerts in non-emergency situations.
| Alert | Probable cause | Probable solution | Can application owner fix this with their current perms? | Should they do this in a non-emergency situation? |
|---|---|---|---|---|
OpenSearchNode(Low,High)DiskWatermarkReached
|
The OpenSearch pods are out of disk space | Expand disk space | Yes | No |
OpenSearchClusterAtLeastOneRedIndex
|
At least one of the indices is missing data. | Delete the broken index/regenerate the data | Yes | Only if the cause is well-understood (I lost data on an index I don't care about, let me completely delete the index to make the alerts go away). |
OpenSearchBulkRequestsRejectionJumps
|
The OpenSearch pods are overwhelmed | 1. Determine cause (increased traffic? dying K8s worker?)
2. Take action based on cause (increase resources, block attacker, etc) |
Unknown | Only if the cause is well-understood (I just tried to ingest a 1 trillion line bulk JSON file into OpenSearch). |
OpenSearchJVMHeapUseHigh
|
The OpenSearch pods are using too much memory |
- Check pod lifetimes to see if the pods are getting frequently OOMKilled - Typical Java memory troubleshooting. - Consider raising memory requests/limits in Kubernetes up to a maximum of 16 GB |
Yes | No |
Operations
Deploying a New OpenSearch on K8s Cluster
- Begin by following the general instructions for deploying a new Kubernetes service.
- Create/merge a patch that adds your new namespace to the list of tenantNamespaces in deployment-charts repo (example CR). OpenSearch uses RBD, so you want to change the
helmfile.d/admin_ng/values/dse-k8s-eqiad/ceph-csi-rbd-values.yamlfile. - After merging, you will need to deploy
admin_ngfor both dse-k8s clusters. - Create/merge a patch that increases default resources for your new namespace (example patch) . This is needed because the version of the operator chart we use (2.7.0) does not allow changing the resources allotted to the bootstrap pod, and it needs at least 2 GB RAM to stand up the cluster.
- Create and add secrets. By default, you will need the following secrets:
| Secret Name | Details |
|---|---|
| username | opensearch |
| password | Used to access the OpenSearch REST API via basic auth. Read/write access to OpenSearch indices only; cannot do cluster operations. |
| hashed_password | Used to populate the opensearch security YAML config, which in turn is pushed to the OpenSearch Security API via securityadmin.sh . This must be a bcrypted hash. You can generate this with htpasswd , Python's bcrypt library, etc.
|
| Secret Name | Details |
|---|---|
| username | opensearch-operator |
| password | Used to access the OpenSearch REST API via basic auth. Used by OpenSearch operator and power users for cluster-level operations |
| hashed_password | Used to populate the opensearch security YAML config, which in turn is pushed to the OpenSearch Security API via securityadmin.sh . This must be a bcrypted hash. You can generate this with htpasswd , Python's bcrypt library, etc.
|
Better User Management is Coming
We should have better user management once this Phab ticket gets some attention.
- Create and merge a patch that tells the OpenSearch operator to watch your new namespace. Example patch
Phases of Deployment
1. The bootstrap, first master, and security pods are created:
NAME READY STATUS RESTARTS AGE
cluster-bootstrap-0 1/1 Running 0 61s
cluster-masters-0 0/1 Running 0 61s
cluster-securityconfig-update-f7wmp 1/1 Running 0 61s
During this phase, the securityconfig pod (which creates the .opendistro_security index, somewhat akin to the MySQL users table) continually polls the cluster, waiting for the cluster to reach yellow status (in this context, when the bootstrap node and initial master node form a cluster). You will see a lot of Waiting to connect to the cluster in the security config pod, this is normal for the first few minutes of a deploy.
If you look at the bootstrap pod's logs, you might see logs complaining that the .opendistro_security index doesn't exist. This is also normal for the first few minutes of a deploy.
 | About the Bootstrap Node The bootstrap node is a full-fledged OpenSearch cluster node, as opposed to a lightweight initContainer that just runs a script. So what makes it different from the other OpenSearch cluster nodes?
(A) It starts before any other pods, I believe this is to prevent race conditions and ensure a predictable cluster formation process. (B) Its resources are set via namespace-specific container defaults rather than directly in the chart (see this CR for an example). The bootstrap pod needs at least 2 GB of RAM to do its job. |
2. The cluster has formed, but not all the masters have deployed:
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
cluster-bootstrap-0 1/1 Running 0 10m
cluster-masters-0 1/1 Running 0 10m
cluster-masters-1 0/1 Running 0 8m
cluster-securityconfig-update-f7wmp 0/1 Completed 0 10m
This phase begins once the securityconfig pod creates the .opendistro_security index. As shown above, this pod's status will change to Completed and the operator will create any additional masters.
3. The cluster is fully deployed:
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
cluster-masters-0 1/1 Running 0 63m
cluster-masters-1 1/1 Running 0 62m
cluster-masters-2 1/1 Running 0 60m
cluster-securityconfig-update-f7wmp 0/1 Completed 0 63m
This is what the cluster looks like after a successful deploy. Note that the operator has removed the bootstrap pod.
Verifying Cluster Access
Start a port-forward in one terminal:
$ kubectl port-forward --address 0.0.0.0 cluster-masters-0 :9200
Forwarding from 0.0.0.0:45679 -> 9200
In a second terminal, set your credentials and port. You can find the username and password in /srv/git/private/hieradata/role/common/deployment_server/kubernetes.yaml on the Puppet server.
$ PW=user:password # used for basic auth with cURL
$ PT=45679 # must match the port from the port-forward above
Test with authentication:
$ curl -ks -u ${PW} https://0:${PT}/_prometheus/metrics
Or without authentication (read-only access is enabled by default):
$ curl -ks https://0:${PT}/_cat/nodes
$ curl -ks https://0:${PT}/_cat/health
$ curl -ks https://0:${PT}/_prometheus/metrics
Deploying on an existing cluster
From the deployment server, run kube_env ${namespace}-deploy dse-k8s-${site} . This sources all the environment variables you need to run helmfile apply, delete pods, etc...
Changing Resources on a Live Cluster
The chart has several places to change requests/limits.
The resource configuration that applies to the cluster nodes is under
opensearchCluster:
nodePools:
- component: masters
roles:
- "master"
- "data"
resources:
requests:
memory: "4Gi"
cpu: "2000m"
limits:
memory: "4Gi"
cpu: "2000m"
As of this writing, that configuration is applied via this file in our deployment-charts repo. Which means if you want to override it, you need to add it to the values.yaml file specific to your deployment and/or helmfile release.
Again, as of this writing, the operator does not detect changes to the resources or source image.
As such, you'll have to delete the pods one-by-one to apply the resource changes. Note that since the PersistentVolumes and their PersistentVolumeClaims are not deleted, the data is not at risk. In practice it feels a lot more like a very quick reboot (delete the pod, the operator replaces it immediately, the new pod is connected to the existing OpenSearch data). It typically only takes a minute or two after deletion for the cluster to get back into its healthy "green" state.
But you should still take your time and ensure the cluster is healthy before moving on to the next pod. You can use the following API calls to check cluster health:
$ curl -H 'Accept:Application/yaml' -XGET https://opensearch-test.discovery.wmnet:30443/_cluster/health
---
cluster_name: "opensearch-test"
status: "yellow"
timed_out: false
number_of_nodes: 3
number_of_data_nodes: 3
discovered_master: true
discovered_cluster_manager: true
active_primary_shards: 8
active_shards: 18
relocating_shards: 0
initializing_shards: 0
unassigned_shards: 2
delayed_unassigned_shards: 0
number_of_pending_tasks: 0
number_of_in_flight_fetch: 0
task_max_waiting_in_queue_millis: 0
active_shards_percent_as_number: 90.0
---
This cluster is yellow. If it stays in yellow for more than a few minutes after launching a new pod, further investigation is required. Did the new pod join the cluster? Are shards moving to it? If there are under-replicated shards, are they on the pod you want to delete?
$ curl -XGET https://opensearch-test.discovery.wmnet:30443/_cat/nodes
10.67.29.29 8 56 1 0.80 1.39 1.72 dm cluster_manager,data - opensearch-test-masters-2
10.67.28.200 36 83 4 0.83 1.13 1.45 dm cluster_manager,data * opensearch-test-masters-0
This cluster is missing a pod, wait for the third pod to finish deploying before deleting another pod.
Example: Rolling Resource Update
1.
$ kubectl get pod -o yaml opensearch-test-masters-0 | grep -i memory
memory: 2Gi
memory: 2Gi
We have applied the change via helmfile, but the operator hasn't detected the changes.
2.
$ kubectl delete pod opensearch-test-masters-0
We've deleted one of the pods to force its replacement.
3.
$ kubectl get pod -o yaml opensearch-test-masters-0 | grep -i memory
memory: 4Gi
memory: 4Gi
After a few minutes, the pod has been replaced and it has the new specs.
The operator should be handling this automatically. We'll continue investigating, but it's likely this will be fixed when we move to a newer version of the operator and/or helm chart.
Renewing TLS Certificates
The current version of the OpenSearch operator does not currently support hot reloading of certificates (See phab:T414217 for tracking automatic certificate reload in a future operator version).
Note that kubectl get cert is NOT sufficient to see which certificate is presented by the cluster, you have to hit the endpoint directly:
# certificate for endpoint
$ kubectl port-forward service/${K8S_NAMESPACE%-deploy}-bulk 9200 &
$ gnutls-cli --print-cert localhost:9200 | grep --line-buffered CN=opensearch-wmf | grep --line-buffered -o 'expires[^,]*'
expires 2026-06-16 02:51:00 UTC
# certificate for opensearch itself
$ kubectl exec -it ${K8S_NAMESPACE%-deploy}-masters-0 -- openssl x509 -enddate -noout -in /usr/share/opensearch/config/tls-http/tls.crt
notAfter=Jun 2 12:29:00 2026 GMT
1) Follow Kubernetes/Kubectl/Cheat Sheet#Force refresh of a cert-manager certificate
$ kube-env ${NAMESPACE}-deploy dse-k8s-eqiad
$ kubectl patch certificate/opensearch-wmf --type=json --patch='[{"op": "replace", "path": "/spec/renewBefore", "value": "1440h"}]'
$ sleep 10
$ kubectl patch certificate/opensearch-wmf --type=json --patch='[{"op": "remove", "path": "/spec/renewBefore"}]'
2) Rolling-restart all pods in the cluster to take this new certificate into account
# on deploy2002.codfw.wmnet
# I suggest running this in a tmux if the cluster is large
$ /home/brouberol/restart-sts-pods -e dse-k8s-eqiad -n ${NAMESPACE} ${NAMESPACE}-masters
$ /home/brouberol/restart-sts-pods -e dse-k8s-eqiad -n ${NAMESPACE} ${NAMESPACE}-data # might not exist
What do to when the pods get stuck
If you're running the script and you see
Readiness: ............................................................................
for more than 5 minutes:
A. Check container status:
kubectl get po | grep -i creat
opensearch-semantic-search-data-1 0/1 ContainerCreating 0 3h22m
If you see a pod stuck in "ContainerCreating", it's likely the physical volume (PV) associated with that pod was never properly detached from the previous pod's host.
B. Find the persistent volume claim (PVC) associated with the pod:
kubectl get pvc | grep opensearch-semantic-search-data-1
data-opensearch-semantic-search-data-1 Bound pvc-54990147-2c50-47b7-9b5a-3d87d3c387e6 170Gi RWO
The third field (pvc-54990147-2c50-47b7-9b5a-3d87d3c387e6 in this example) is the PV ID. Copy it to your clipboard.
C. Figure out which k8s worker is holding on to the PV:
Login to a cumin host and check all k8s workers in the datacenter for lingering mounts:
sudo cumin A:dse-k8s-worker-eqiad 'mount | grep pvc-54990147-2c50-47b7-9b5a-3d87d3c387e6'
24 hosts will be targeted:
dse-k8s-worker[1001-1019,1024-1028].eqiad.wmnet
OK to proceed on 24 hosts? Enter the number of affected hosts to confirm or "q" to quit: 24
===== NODE GROUP =====
(1) dse-k8s-worker1025.eqiad.wmnet
----- OUTPUT of 'mount | grep pvc...b5a-3d87d3c387e6' -----
/dev/rbd4 on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-54990147-2c50-47b7-9b5a-3d87d3c387e6/globalmount/0001-0024-6d4278e1-ea45-4d29-86fe-85b44c150813-0000000000000007-a659f72b-1e1f-11f1-be9c-5a028bb344e6 type ext4 (rw,relatime,stripe=1024,_netdev)
In the above example, dse-k8s-worker1025.eqiad.wmnet is the host that's holding onto the mount. Let's go there next.
D. Unmount the PV manually from the k8s worker:
#umount /dev/rbd4
It will take 5-10 minutes for the PV to be completely freed up, but that should do it.
Note that you can also delete the pod and its PVC to get the same effect, but this is not recommended because:
- It completely wipes out the data for that pod. OpenSearch is self-healing, so it will recover the data from other pods. But it's better not to waste resources on the recovery process if we don't have to.
- The PV will not be deleted. It's hard to find these unused PVs (FIXME:add HOWTO for finding them) and they waste space on the Ceph cluster.
3) Verify that your endpoint is serving the new certificate:
$ gnutls-cli --print-cert ${NAMESPACE}.svc.eqiad.wmnet:30443 | grep expire
- subject `CN=opensearch-wmf', issuer `CN=discovery,OU=SRE Foundations,O=Wikimedia Foundation\, Inc,L=San Francisco,C=US', serial 0x69ad031ab6ce85e4abf0a1be545c95e0d716daef, RSA key 2048 bits, signed using ECDSA-SHA512, activated `2026-02-05 19:35:00 UTC', expires `2026-03-05 19:35:00 UTC', pin-sha256="AcCR/RPH1ogjJfEnPi91ltR4uCLEY7bFtwcVIgSFfCU="
4) Confirm that all certificate alerts have cleared. The Blackbox probes that monitor certificate expiration come from outside K8s, so they could be hitting any one of the pods. If the alerts don't all clear, you may have forgotten to cycle through all of the pods.
Completely Wiping Out/Redeploying an OpenSearch on K8s Cluster
If you need to wipe out and redeploy your cluster, here are the steps:
- Login to the deploy host
- Become the -deploy user:
kube_env ${namespace}-deploy dse-k8s-${site}
- Navigate to the correct
helmfile.ddirectory:cd /srv/deployment-charts/helmfile.d/dse-k8s-services/${namespace}
- Destroy using helmfile command
helmfile -e dse-k8s-${site} -i destroy
| If you stop here, you won't be able to redeploy your cluster! helmfile destroy does not touch the Kubernetes Persistent Volume Claims (PVCs). If you don't delete the PVCs, your next deploy will fail because OpenSearch will attach to the existing PVCs, recognize the old cluster you destroyed, and refuse to create a new cluster. |
- Check for/destroy unused PVCs:
1.
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-opensearch-semantic-search-masters-0 Bound pvc-fc747e0f-cafc-40c4-8345-bbf7abe88f3c 40Gi RWO ceph-rbd-ssd 21d
data-opensearch-semantic-search-masters-1 Bound pvc-fde85989-6cef-4021-b3e7-10e362f3f997 40Gi RWO ceph-rbd-ssd 21d
data-opensearch-semantic-search-masters-2 Bound pvc-497f0df2-22a5-4330-b9a6-9d78629f4593 40Gi RWO ceph-rbd-ssd 21d
data-opensearch-semantic-search-masters-3 Bound pvc-4fe56d26-2613-4936-9cb3-4ff9cf41279c 30Gi RWO ceph-rbd-ssd 154m
data-opensearch-semantic-search-masters-4 Bound pvc-9d3f9612-0230-43e8-b685-452ad055817e 30Gi RWO ceph-rbd-ssd 153m
data-opensearch-semantic-search-masters-5 Bound pvc-18431dad-59c1-42d6-bdf7-cc34e3ef50d4 30Gi RWO ceph-rbd-ssd 151m
data-opensearch-semantic-search-masters-6 Bound pvc-6e87fe5f-b966-4662-af9e-c0fc7f537de4 30Gi RWO ceph-rbd-ssd 150m
2.
for n in {0..6}; do kubectl delete pvc data-${namespace}-masters-${n}; done
3. Verify things are completely gone:
kubectl get all
kubectl get pvc
After that, you should be clear to redeploy.
API Calls
Audit Logging
PT=43885; curl -H 'Content-type: Application/json' -XPOST -k -u ${PW} https://0:${PT}/security-auditlog-2025.10.31/_search?pretty
Forcing Shard Reallocation
! You will need the `operator` OpenSearch user as opposed to the typical `opensearch` user!
Dynamic and Static Configuration settings
| Aspect | Static | Dynamic |
|---|---|---|
| Where to set | opensearch.yml. If using the opensearch-cluster helm chart, additionalConfig is added to opensearch.yml |
set via the OpenSearch API |
| Activating | You have to restart the OpenSearch service. Theoretically, this should be handled by the operator, but you may need to rollout restart to apply the settings if the settings don't change |
Settings are active as soon as OpenSearch accepts your API call |
| How to tell if setting is known to OpenSearch | You will NOT see the changes in configMaps or inside the pod's opensearch.yml, as the operator adds new settings as environment variables. You can run env inside the pod to check if the setting exists. | Check via OpenSearch API |
| How to tell if setting is applied | OpenSearch config follows Postel's law, and it will accept invalid config (For example, see this change where I added config that OpenSearch accepted, but never really worked). You may need to do more investigation to verify the setting is:
1) correct and 2)active. OpenSearch will ignore some settings even when they are correct and active if they would have detrimental effects on the cluster (for example, it will allow shards to land on a banned node if there is no other place to put them). |
Same |
Disk Usage
How to Check
By default, the OpenSearch on K8s pods have 30 GB disks. You can check disk usage via the following API call:
$ curl -s https://opensearch-ipoid.svc.eqiad.wmnet:30443/_cat/allocation?v
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
21 8.3gb 8.3gb 20.9gb 29.3gb 28 opensearch-ipoid-masters-2 10.67.28.219 opensearch-ipoid-masters-2
21 4.6gb 4.6gb 24.6gb 29.3gb 15 opensearch-ipoid-masters-0 10.67.26.116 opensearch-ipoid-masters-0
22 13gb 13gb 16.2gb 29.3gb 44 opensearch-ipoid-masters-1 10.67.28.6 opensearch-ipoid-masters-1
Disk usage is also visible via this panel from the OpenSearch on K8s dashboard
How to Expand
1. Merge a change that raises the disk size. Note that in the current version of the opensearch-cluster helm chart (2.7.0), you will have to override the entire nodePools object within the opensearchCluster object, see this CR for an example.
2. Apply the change using our standard deployment process (helmfile apply).
3. The change should be visible in OpenSearch immediately. Verify by using the same call as above:
$ curl -s https://opensearch-ipoid.svc.eqiad.wmnet:30443/_cat/allocation?v
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
25 6.4gb 6.4gb 32.7gb 39.2gb 16 opensearch-ipoid-masters-0 10.67.29.37 opensearch-ipoid-masters-0
24 4.6gb 4.6gb 34.5gb 39.2gb 11 opensearch-ipoid-masters-2 10.67.30.227 opensearch-ipoid-masters-2
24 12.8gb 12.8gb 26.3gb 39.2gb 32 opensearch-ipoid-masters-1 10.67.28.5 opensearch-ipoid-masters-1
Now we have ~40GB of disk space per pod instead of the previous ~30GB!
OpenSearch versions
Current versions
See our Docker images repo for the available versions.
Version update cadence
The platform owners will update the OpenSearch version at least once per quarter. Since our docker images are sourced from upstream Debian packages, we can only update as far as what's available there.
Application owners are welcome to request newer versions at any time.
OpenSearch Plugins
If your application requires a plugin that's not in the default OpenSearch install, open a Phab task to request the plugin. Plugins will be evaluated on a case-by-case basis. Criteria to be evaluated:
- Whether or not the plugin is built by the OpenSearch team
- Whether or not the plugin is owned by WMF
- Security
- Resources consumed by plugin
- Responsiveness of plugin developers
You can see a list of existing plugins by looking for the phrase opensearch-plugin install in the Blubberfile that creates the Docker image for your OpenSearch version (example ). FIXME: improve discoverability of installed plugins
Updating the OpenSearch Version
The OpenSearch image is built from Debian packages. We mirror the upstream packages in repo components such as
thirdparty/opensearch3 trixie-wikimedia
Before we can update the OpenSearch docker image version, we'll need to update the OpenSearch package version we mirror.
From the apt servers, check for updates. Example command:
$ reprepro --component thirdparty/opensearch3 checkupdate trixie-wikimedia
If updates are available, you'll see something like:
Calculating packages to get...
Updates needed for 'trixie-wikimedia|thirdparty/opensearch3|amd64':
'opensearch': '3.3.2' will be upgraded to '3.5.0' (from 'thirdparty/opensearch3'):
files needed: pool/thirdparty/opensearch3/o/opensearch/opensearch_3.5.0_amd64.deb
Note that reprepro only keeps a single version of the package. Thus, after you update the package version, you will need to immediately update the Docker image repo with new version of the package, including all plugins and their checksums.
reprepro does support rolling back to an older package version if things go wrong, assuming that the package is still available from upstream.
Miscellenia
Block device optimizations
Per phab:T419041 , we have a script that sets /sys/block/rbd[0-9]*/queue/read_ahead_kb on all OpenSearch pods. This script run on all dse k8s workers and is applied every 5 minutes via a systemd timer. The script requires `crictl` to function.
Troubleshooting
Check the journal on the dse-k8s-worker:
journalctl -u set-rbd-readahead.service
Mar 24 19:15:00 dse-k8s-worker2001 set-rbd-readahead.py[922404]: 2026-03-24 19:15:00,455 INFO rbd5 read_ahead_kb already set to 64, not making changes...
To prove it's making changes:
journalctl -u set-rbd-readahead.service --since '1 day ago' | grep Set | egrep -v "(Starting|Finished)"
Mar 23 19:40:01 dse-k8s-worker2001 set-rbd-readahead.py[3605610]: 2026-03-23 19:40:01,014 INFO Set rbd3 read_ahead_kb -> 64
Mar 23 19:50:01 dse-k8s-worker2001 set-rbd-readahead.py[3616266]: 2026-03-23 19:50:01,010 INFO Set rbd5 read_ahead_kb -> 64
Mar 23 20:05:00 dse-k8s-worker2001 set-rbd-readahead.py[3633968]: 2026-03-23 20:05:00,994 INFO Set rbd6 read_ahead_kb -> 64
FIXME: Improve logging to show prior read_ahead_kb setting and make it easier to follow which pods are associated with the RBD device.