Data Platform/Systems/OpenSearch-on-K8s
You've reached the documentation for the OpenSearch on K8s platform!
Intended Audience
Foundation employees who need an OpenSearch backing store for their application (small-to-medium use cases only), or who are already building on the platform.
SREs who support the platform may be more interested in this page.
Service/Availability Expectations
The OpenSearch on K8s platform is supported on a “best effort” basis, which means the platform owner (DPE SRE) will work to restore the platform during business hours only.
Alerts, notifications, and responders will be the same as other DPE SRE-supported services, although it is possible to customise the monitors and recipients for the specific OpenSearch cluster workload.
Responsibilities for the individual OpenSearch cluster are split between the platform owner and application owner, as defined below.
Glossary
| Term | Definition |
|---|---|
| Platform owner | The team that owns OpenSearch on Kubernetes (currently Data Platform SRE) |
| Application owner | The team or person who owns an application which uses OpenSearch on Kubernetes as a backing store. |
Responsibilities
| Action | Responsibility |
|---|---|
| Restore Kubernetes services | platform owner |
| Restore OpenSearch operator service | platform owner |
| Redeploy an OpenSearch cluster | platform owner |
| Load data into the OpenSearch cluster | application owner |
| Ensure application can withstand the loss at least one of its availability zones(eqiad/codfw datacenters) | application owner |
| Regenerate lost OpenSearch data | application owner |
| Index settings (replica/shard counts) | application owner† |
| Index state management (pruning/deleting old indices) | application owner† |
† See "OpenSearch best practices" section below.
Service Levels
The Wikimedia Foundation has adopted Service level objectives (SLOs) as its primary tools for measuring reliability and tuning our priorities accordingly.
Please see the page SLO for more general information about the adoption of these principles across the Foundation, particularly the definitions for service level indicators and service level objectives.
At the platform level, our OpenSearch on Kubernetes services offer a low ceiling availability SLO of 95%. This reflects our the fact that DPE SRE does not provide 24/7 support. Essentially, OpenSearch can fail on a Friday afternoon and it would be reasonable to have to wait until Monday morning before service is restored. Applications built on top of the platform must take this into account when defining their SLOs and may wish to include graceful degradation capability, if appropriate.
Service Level indicators (SLIs)
Our platform provides standard SLIs on availability and latency for each OpenSearch cluster. We have certain metrics that are generated from all opensearch clusters, which can be used to craft the service level indicators for your application. These SLIs may be tuned to a specific application and additional SLIs may be included from other systems and subsystems, where appropriate.
For technical reasons, these standard availability and latency measurements are made at the boundary where requests ingress to the cluster. This will capture all client application requests, whether they come from inside or outside the Kubernetes cluster environment. However, we do not yet provide any SLIs for cluster-internal HTTP/HTTPS traffic, since this does not cross the boundary where we take measurements.
The standard availability SLI is defined as:
- The percentage of queries that return a successful HTTP status code (2xx), excluding 4xx client errors.
This standard indicator is stated in such as way that it includes all API calls, thereby capturing both search and bulk index queries, as well as all other OpenSearch API calls.
The standard latency SLI is defined as:
- The percentage of search requests served in under n milliseconds.
In this case, the value of n may be adjusted depending on the application. For example, an application that only makes very simple queries against a relatively static index might have a lower latency expectation than an application that makes complex queries against an index that is being continuously indexed. A value of 100 ms might be useful for a user-interactive search-heavy system.
This standard SLI also states that it is search request latency, that is measured by this indicator. We specifically exclude the Bulk APIand the Streaming bulk API calls from this metric, since they naturally have a much greater response time. The bulk index API latency (and availability) metrics are available, so if these would be important to your application, then an SLI can be defined from them
Service Level Objectives (SLOs)
A set of SLOs will be defined for each application hosted on our OpenSearch on Kubernetes platform. The user requirements and technical requirements are application specific, so a fully standard set of SLOs to cover all applications would not be beneficial.
SLO/OpenSearch IPoid is our first application that has a defined SLO for its OpenSearch component. This application uses our standard SLIs to define SLOs for its availability and latency. This application also has an SLO for data freshness, which is based on the elapsed time since the Airflow job that loads the data last succeeded, in both data centres.
As an example, the SLOs for this OpenSearch IPoid service can be stated as:
- 95% of queries return a successful HTTP status code (2xx), excluding 4xx client errors.
- 95% of search requests served in under 300 milliseconds.
- Data is no older than 72 hours.
Once the SLOs for an OpenSearch on Kubernetes cluster have been defined and ratified (taking into account the 95% ceiling for availability of the platform) they can be added to the slothslos git repository, in code form.
This will generate the required dashboards to allow monitoring of the SLO and alerts to notify of potential SLO violations and error budget burning.
If you have any queries around SLIs or SLOs for a potential OpenSearch on Kubernetes use-case, please don't hesitate to get in touch with the DPE SRE team to discuss them.
Disaster Recovery/Return to Service
The OpenSearch service shall not be the primary backing source of data for any application. In case of data loss, the application owner is expected to regenerate/reload the data into their OpenSearch on Kubernetes instance. OpenSearch-native backups and restores are not part of the platform at this time.
Support Escalations
If you encounter a situation that can't be solved by the troubleshooting section below, not to worry! You can contact DPE SRE via #data-platform-sre Slack (preferred) or #wikimedia-data-platform IRC.
Lifecycle Work/Upgrades
The physical infrastructure which hosts OpenSearch on Kubernetes will need to go offline from time to time. When possible, the platform owner shall notify the application owners via email at least 5 business days before any planned service interruption.
Security Updates
In case of security updates, the platform owners will redeploy the service. When possible, the platform owner shall notify the application owners via email at least 5 business days before any planned service interruption.
Responsible Usage Policy
In the event that the platform owners reasonably suspect that platform-wide issues are being caused by a specific application, the platform owners may take steps to ensure the stability of the platform. These actions include (but are not limited to): rate-limiting, changing resources/limits, temporarily or permanently disabling the OpenSearch cluster. The platform owners will notify the application owner and involve them in the mitigation process as much as is practical.
Consultations
Users can request general advice on OpenSearch optimizations, workflows, etc from DPE SRE and other subject matter experts at WMF (such as the Search Platform team). Consultations are subject to SME availability and are not considered to be part of the SLO. OpenSearch query optimization is one example of a consultation that would not fall under the SLO.
Platform Info
Design
OpenSearch on Kubernetes runs in WMF’s primary datacenters (eqiad and codfw). Each datacenter’s service will be independent (data will not be replicated between datacenters).
The platform owners can offer assistance and advice on multi-DC configuration (code review, failover tests, etc) but the application owner is responsible for ensuring the application can survive the loss of a single primary DC (if so desired by the application owner; this is not a requirement for running on the platform).
Flavors
Below is the "t-shirt size" breakdown of available resources. New clusters will get size opensearch-small unless otherwise requested.
| Name | Number of replicas | CPU cores/replica | RAM/replica (GB) | default disk space/replica (GB) | maximum disk space/replica (GB) |
|---|---|---|---|---|---|
| opensearch-small | 3 | 2 | 4 | 30 | 100 |
| opensearch-medium | 3 | 4 | 8 | 30 | 100 |
| opensearch-large | 3 | 8 | 16 | 30 | 100 |
OpenSearch versions
Current versions
See our Docker images repo for the available versions.
Version update cadence
The platform owners will update the OpenSearch version at least once per quarter. Since our docker images are sourced from upstream Debian packages, we can only update as far as what's available there.
Application owners are welcome to request newer versions at any time.
OpenSearch Plugins
If your application requires a plugin that's not in the default OpenSearch install, open a Phab task to request the plugin. Plugins will be evaluated on a case-by-case basis. Criteria to be evaluated:
- Whether or not the plugin is built by the OpenSearch team
- Whether or not the plugin is owned by WMF
- Security
- Resources consumed by plugin
- Responsiveness of plugin developers
You can see a list of existing plugins by looking for the phrase opensearch-plugin install in the Blubberfile that creates the Docker image for your OpenSearch version (example ). FIXME: improve discoverability of installed plugins
Getting Started With Your New Cluster
Access
Read-only anonymous access
You can reach your new cluster at https://${kubernetes-namespace}.discovery.wmnet:30443/ from anywhere inside WMF.
For example, the opensearch-test cluster URL is
https://opensearch-test.discovery.wmnet:30443/
If you asked for a multi-DC deploy, your cluster has endpoints at
https://${kubernetes-namespace}.svc.${dc}.wmnet:30443
as well.
Write access
Application owner
In order to write to your cluster, you have to supply a username and password. The default username is opensearch. Each deployment has its own password, which can be found on the deployment server via the following one-liner:
kube_env ${kubernetes_namespace}-deploy dse-k8s-${dc} ; kubectl get secret ${kubernetes_namespace}-admin-password -o jsonpath='{.data.password}' |base64 -d
Platform owner
SREs needing to perform some cluster operations (such as forcing shard reallocation) will need to use the operator user until https://phabricator.wikimedia.org/T416714 is complete.
Troubleshooting
Dashboards and Alerts
Dashboards
See our OpenSearch on K8s dashboard. The dashboard contains a number of metrics we use to gauge health.
Look out for the 'interval' dropdown
Unlike other WMF dashboard, OpenSearch on K8s contains an "interval" dropdown, used to get a rolling average. If the panels are showing up blank, try to lengthen the interval. But you should also realize if you make the interval too long, it might confuse things as well.
For example, we had an incident where latency spiked to about 10x its normal value. After the incident was resolved, we still had the interval set to 24h. Thus, it was still including periods of extremely high latency in its graphs, making it seem like the incident was still ongoing. We are working to address this confusion in in T417230 .
Important Dashboard Panels
- The "network probes" panel, showing the latency of all OpenSearch clusters per DC.
- Disk Usage - Nothing good happens when you run out of disk space!
- Memory usage - amount of memory used per pod
- Search latency
You can find alerts in the Wikimedia Foundation's alert repo .
Alerting
These alerts will notify DPE SRE once your cluster is in "production" state in service.yaml in Puppet.
As stated in the "Service Availability/Expectations" section, DPE SRE will respond to the alerts within business hours.
The table below describes whether or not the application owner has permissions to fix the problem. This is not meant to imply the application owner must fix the problem. Generally speaking, the platform owner (DPE SRE) is the right team to deal with alerts in non-emergency situations.
| Alert | Probable cause | Probable solution | Can application owner fix this with their current perms? | Should they do this in a non-emergency situation? |
|---|---|---|---|---|
OpenSearchNode(Low,High)DiskWatermarkReached
|
The OpenSearch pods are out of disk space | Expand disk space | Yes | No |
OpenSearchClusterAtLeastOneRedIndex
|
At least one of the indices is missing data. | Delete the broken index/regenerate the data | Yes | Only if the cause is well-understood (I lost data on an index I don't care about, let me completely delete the index to make the alerts go away). |
OpenSearchBulkRequestsRejectionJumps
|
The OpenSearch pods are overwhelmed | 1. Determine cause (increased traffic? dying K8s worker?)
2. Take action based on cause (increase resources, block attacker, etc) |
Unknown | Only if the cause is well-understood (I just tried to ingest a 1 trillion line bulk JSON file into OpenSearch). |
OpenSearchJVMHeapUseHigh
|
The OpenSearch pods are using too much memory |
- Check pod lifetimes to see if the pods are getting frequently OOMKilled - Typical Java memory troubleshooting. - Consider raising memory requests/limits in Kubernetes up to a maximum of 16 GB |
Yes | No |
OpenSearch Best Practices
- OpenSearch indices default to 1 replica and 1 primary. Since OpenSearch on K8s deployments have 3 nodes, create your indices with 2 replicas instead of 1. This will give you better read performance and more redundancy. To set this cluster-wide for any newly-created indices (highly recommended), use the Cluster Dynamic Setting
cluster.default_number_of_replicas. To apply to an existing index or while creating a new index, set index update settingindex.number_of_replicas.
- Setting up aliases will simplify future changes, when wanting to introduce changes in mappings, or in the number of shards, which cannot be done on a live index.
- Likewise, you should use Index State Management to manage the lifecycle of your indices (such as periodically reducing replica count or deleting older indices). Note that the OpenSearch Dashboards option mentioned in the link is not available in the OpenSearch on K8s platform, so you will need to use the API.
- There is no inter data center replication. Client applications should send writes to both clusters.