SLO/OpenSearch IPoid

Status: draft

Organizational

Service

OpenSearch IPoid provides an API for retrieving reputation data for IP addresses (e.g., identifying VPNs, proxies, or residential IPs). It serves as the storage and retrieval backend for the IPReputation extension, which is also used by mw:Extension:IPInfo and mw:Extension:WikimediaEvents. The service uses OpenSearch to index and query data imported from Spur.

Teams

Service Owner: Product Safety and Integrity (PSI) – responsible for the data ingestion logic in Airflow, the IPReputation and IPInfo extensions, and feature maintenance.
Infrastructure Owner: Search Platform (Data Platform SRE) – responsible for the underlying OpenSearch cluster on Kubernetes (dse-k8s).

Architectural

Environmental dependencies

The service runs as an OpenSearch cluster within the dse-k8s Kubernetes environment (currently in eqiad and codfw).

Namespace: opensearch-ipoid
Storage: Data Platform/Systems/Ceph

Service dependencies

Spur Data Feed (Hard Dependency): The service relies on daily updates from Spur. If ingestion fails, data becomes stale, though read availability remains intact. There is a daily cleanup job that removes entries older than 7 days. So if feed ingestion fails, there is a one week period to fix this before all data is unavailable.
OpenSearch Operator: Manages the lifecycle of the OpenSearch cluster in Kubernetes. Note that downtime for the opensearch-operator will not immediately affect the cluster resources, only lifecyle coperations.
Ceph CSI RBD Plugin: The ceph-csi-rbd plugin for kubernetes is used to allocate block devices for the opensearch pods. Similarly, the plugin is only used to provision block devices, so downtime of the plugin will not cause an immediate failure of the opensearch-ipoid cluster. Only operations that cause the opensearch pods to be recreated would cease to function.
Ceph: The Ceph clusters operated by the Data Platform Engineering team are a hard dependency. Without this service running, the OpenSearch pods will fail to read the indices from their virtual disks and the update task will not be able to update the indices.
Airflow: Orchestrates the jobs that fetch data from Spur and index it into OpenSearch.

Client-facing

Clients

MediaWiki (IPReputation Extension): The primary consumer. MediaWiki makes synchronous requests to OpenSearch IPoid when privileged users view user information or Special:IPInfo.
SRE/Admin Tools: Occasional ad-hoc queries for investigation.

Request Classes

Query: Read-only lookups for IP reputation data.
Ingestion: Write requests during the daily data update window.

Service Level Indicators (SLIs)

Availability: The percentage of queries that return a successful HTTP status code (2xx), excluding 4xx client errors.
- This will be measured at the istio ingressgateway.
- Promql for errors and total availability values:

error_query: |
  sum(
    rate(istio_requests_total{
      source_workload_namespace="istio-system",
      app="istio-ingressgateway",
      destination_service_namespace="opensearch-ipoid",
      response_code!~"(2|4).."
    }[{{.window}}])
  )
total_query: |
  sum (
    rate(istio_requests_total{
      source_workload_namespace="istio-system",
      app="istio-ingressgateway",
      destination_service_namespace="opensearch-ipoid",
      response_code!~"4.."
    }[{{.window}}])
  ) > 0

Latency: The percentage of non-bulk index requests served in under 500 ms.
- This will be measured at the istio ingressgateway.
- It is important to filter this so that the bulk ingestion API requests are not included.
- Promql for errors and total latency values:

total_query: |
  sum(rate(istio_request_duration_milliseconds_count{
    source_workload_namespace="istio-system",
    app="istio-ingressgateway",
    destination_service_namespace="opensearch-ipoid",
    destination_service_name!="opensearch-ipoid-bulk",
  }[{{.window}}])) > 0
error_query: |
  (
    sum(rate(istio_request_duration_milliseconds_count{
      source_workload_namespace="istio-system",
      app="istio-ingressgateway",
      destination_service_namespace="opensearch-ipoid",
      destination_service_name!="opensearch-ipoid-bulk",
    }[{{.window}}]))
    -
    sum(rate(istio_request_duration_milliseconds_bucket{
      source_workload_namespace="istio-system",
      app="istio-ingressgateway",
      destination_service_namespace="opensearch-ipoid",
      destination_service_name!="opensearch-ipoid-bulk",
      le="500"
    }[{{.window}}]))
  )

We have decided not to implement the freshness SLO using the metric identified below. The reason being that the binary fresh/stale will not work nicely with our tooling, so we will need to instrument the DAG more in order to get a time series for freshness, instead. The information below is retained for possible future reference.

Freshness: The Airflow task that downloads and indexes the spur data task has been successfully executed in both data centres within the last 24 hours.
- At present, the only metric we readily have available for this is the count of successful task instances for this DAG and this task name.
- This means that we cannot easily use a timestamp for this, but we can generate a boolean value if we have not had a successful run within a certain time period.
- Suggested promql query:
```
(
  sum(increase(airflow_ti_finish{
    dag_id="spur_download_and_index_anonymous_residential_eqiad",
    task_id="download_and_index_feed_eqiad",
    state="success"
  }[72h])) > bool 0
)
+
(
  sum(increase(airflow_ti_finish{
    dag_id="spur_download_and_index_anonymous_residential_codfw",
    task_id="download_and_index_feed_codfw",
    state="success"
  }[72h])) > bool 0
)
== bool 2
```
- We could instrument the DAG code further to generate precise timings for these tasks, but we can already use this boolean value to create the required SLO.

Operational

Monitoring

The service is monitored via the standard Prometheus/Grafana stack for dse-k8s and OpenSearch.

Dashboards:
- Miscellaneous TSP metrics (Refer to the IPReputation row)
- OpenSearch on k8s - opensearch-ipoid cluster - codfw
- OpenSearch on k8s - opensearch-ipoid cluster - eqiad
Alerts:
- Email alerts regarding spur.us job failure from Airflow.
- k8s deploymentUnavailable alerts for opensearch-ipoid.
- k8s-ceph RBD volume capacity alertsfor OpenSearch.
- Generic opensearch on k8s alerts.

Troubleshooting

Follow OpenSearch on k8s guidance at: Data Platform/Systems/OpenSearch-on-K8s/Administration
Connectivity issues: Check dse-k8s ingress logs and Network Policies.
Data issues: Verify Airflow job status and OpenSearch index stats (_cat/indices).
Cluster health: Standard OpenSearch diagnostic commands (_cluster/health, _cat/nodes).

Deployment

Infrastructure: Deployed via Helm charts and the OpenSearch Operator, managed by the Data Platform SRE Platform team.
Configuration: Updates are applied via Kubernetes manifests in the relevant deployment repository.
Data Updates: Automated daily via Airflow pipelines.

Service Level Objectives

Grafana dashboard

Realistic targets

Availability: 95%
- The Data Platform SRE team does not operate an out-of-hours on-call rotation, so support for the opensearch clusters, plus the kubernetes and ceph clusters on which they are hosted, may be limited to working hours.
Latency: 95% of search requests served in < 500ms

Ideal targets

Availability: 99.9%
Latency: 99% of search requests served in < 100ms.
Freshness: Data is no older than 24 hours.