SLO/Search

Status: draft

Organizational

Service

The various search components allow full-text search of Wiki articles as well as article title autocompletion and MediaSearch

Teams

Search Platform, a member of the Data Platforms working group, owns these services. The operational side is specifically owned by Data Platforms SRE.

Architectural

Environmental dependencies

CirrusSearch is a MediaWiki extension that handles cross-communication between MediaWiki and Search Elasticsearch clusters.

Our search Elasticsearch clusters are ultimately responsible for serving the search queries forwarded by MediaWiki.

SLIs based around time to render clientside are dependent on the UI; performance regressions in the UI can lead to higher p95 times.

Service dependencies

MediaWiki depends on CirrusSearch being operational.

CirrusSearch depends on the underlying Elasticsearch clusters being healthy and responsive.

Client-facing

Clients

Users submit queries on a given wiki. That wiki's MediaWiki processes generate Elasticsearch queries and, having received the result, display them through the UI.

Request Classes

There are various query types: full text, autocomplete, more_like.

There are also mediasearch requests as well as search preview requests.

Service Level Indicators (SLIs)

All Search SLIs use a Latency SLI, defined as % of all requests that complete within X ms.

Full text p95 latency: <4000 ms

This represents the p95 latency of the time to render the results of a "normal" (full text) search query made via Special:Search (here's enwiki for example)

Autocomplete p95 latency: <600 ms

This represents the p95 latency of the time to render the results of an autocomplete query on the user's screen, also known as a CompSuggest query. This query type is used by Mediawiki and the Wikipedia UI to surface articles whose titles fuzzy match the text of the CompSuggest query.

Example of Wikipedia UI surfacing articles whose title prefixes fuzzy match the query text

Search preview p95 latency: <10000 ms

This represents the p95 latency of the time to render a "flyout" UI element corresponding to a search preview request.

MediaSearch p95 latency: <10000 ms

This represents the p95 latency of MediaSearch queries.

(TODO) Availability: Some notion of the actual availability of Elasticsearch to respond to MediaWiki (the CirrusSearch extension). We can likely use Cirrus failed events (graph) to capture errors generated by lack of service availability as opposed to rejections which correlate with users sending improper queries.

Operational

Monitoring

We get various graphite-based metrics that are plumbed through MediaWiki. These are the major metrics used as SLIs for this service.

Eventually we'll transition to prometheus-based metrics once MediaWiki has been cut over from Graphite to Prometheus.

Troubleshooting

Some issues, like the backend Elasticsearch clusters being temporarily overloaded, are quite simple. Other issues can require more extensive troubleshooting.

Deployment

The CirrusSearch extension for MediaWiki handles all the logic of communication between mediawiki and Elasticsearch.

Additionally, we have 2 "cirrus clusters", in eqiad and codfw respectively, each of which is composed of 3 different Elasticsearch clusters (the multiple ES clusters are for performance reasons; having all of our indices/shards in a single cluster results in poor performance due to Elasticsearch performance being inversely related to total number of shards (after some threshold has been passed).

Service Level Objectives

Realistic targets

Realistic (perhaps overly lenient; we can tighten the non-mediasearch SLOs in future when we have more data) targets:

Full text p95 latency: <4000 ms 95% of the time

Autocomplete p95 latency: <600 ms 95% of the time

Search preview p95 latency: <10000 ms 95% of the time

MediaSearch p95 latency: <10000 ms 95% of the time

Ideal targets

The ideal targets for the SLOs are as follows:

Full text p95 latency: <4000 ms 99.9% of the time

Autocomplete p95 latency: <600 ms 99.9% of the time

Search preview p95 latency: <10000 ms 99.9% of the time

MediaSearch p95 latency: <10000 ms 95% of the time

Reconciliation

Currently we're using the realistic targets because our metrics have only been added in the last several months; once we have more data we should circle back and look if we want to tighten our SLOs accordingly.