Jump to content

Search

From Wikitech
CirrusSearch components diagram
CirrusSearch components diagram

This page covers the usage of Elastica, CirrusSearch, and Elasticsearch, in Wikimedia projects.

In case of emergency

Here is the (sorted) emergency contact list for Search issues on the WMF production cluster:

  • David Causse (dcausse)
  • Erik Bernhardson (ebernhardson)
  • Ryan Kemper (ryankemper)
  • Guillaume Lederrey (gehel)
  • Brian King (inflatador)

You can get contact information from officewiki.

Overview

This system has four components: Elastica, CirrusSearch, the Streaming Updater, and Elasticsearch.

If you want to extend the data that is available to CirrusSearch, have a look at Search/TechnicalInteractionsWithSearch.

Elastica

Elastica is a MediaWiki extension that provides the library to interface with Elasticsearch. It wraps the Elastica library. It has no configuration.

CirrusSearch

CirrusSearch is a MediaWiki extension that provides search support backed by Elasticsearch.

Configuration

The canonical location of the configuration documentation is in the docs/settings.txt file in the extension source. It also contains the defaults, but the source of truth for defaults is extension.json. A pool counter configuration example lives in the README in the extension source.

WMF configuration overrides live in the ext-CirrusSearch.php and CirrusSearch-{common|production|labs}.php files in the mediawiki-config git repo.

Local Build

A dockerized Cirrus environment with cirrus, elasticsearch, and related bits can be sourced from our integration test runner.

Logging

Logs from CirrusSearch can be found from the general Mediawiki logstash dashboard . You can filter with `channel:CirrusSearch AND "backend error"`. This isn't as specific as we'd like it to be, but it should be enough to help get you started.

Elasticsearch

Install

To install Elasticsearch on a node, use role::elasticsearch::server in puppet. This will install the elasticsearch Debian package and setup the service and appropriate monitoring. It should automatically join the cluster.

Configuration

role::elasticsearch::server has no configuration.

In labs a config parameter is required. It is called elasticsearch::cluster_name and names the cluster that the machine should join.

role::elasticsearch::server use ::elasticsearch to install elasticsearch with the following configuration:

Environment Memory Cluster Name Servers
labs 2G configured
beta 4G beta-search Found in hieradata/cloud/eqiad1/deployment-prep/common.yaml in the puppet repo
eqiad 30G production-search-eqiad elastic10([012][0-9]*or*3[01]) (eqiad)
codfw 30G production-search-codfw elastic20(0[0-9|1[0-9]|2[0-4])

Note that the multicast group and cluster name are different in different production sites. This is because the network is flat from the perspective of Elasticsearch's multicast and it is way easier to reserve separate IPs than it is to keep up with multicast ttl.

Files
/etc/default/elasticsearch Simple settings that must be initialized either on the command line (memory) or really early (memlockall) in execution
/etc/elasticsearch/logging.yml Logging configuration
/etc/logrotate.d/elasticsearch Log rolling managed by logrotate
/etc/elasticsearch/elasticsearch.yml Everything not covered above, e.g. cluster name, multicast group
Puppet

The elasticsearch cluster is provisioned via the elasticsearch module of the operations/puppet repository. Configuration for specific machines is in the hieradata folder of the same operations/puppet repository. A few of the relevant files are linked here, but a grep for 'elasticsearch' in hieradata and poking around the elasticsearch puppet module are the best ways to understand the current configuration.

Provisioning

Configuration

  • Common configuration for all elasticsearch servers (this includes the separate elasticsearch cluster used with the ELK stack, not covered in this document).
  • Common configuration for all eqiad elasticsearch servers.
  • regex.yaml - Contains rack identifiers for each elasticsearch server. This is fed into elasticsearch.yml for rack aware shard distribution.
  • Master eligibility is applied to individual selected hosts.
  • A new elasticsearch server will serve queries as soon as it joins the cluster. It needs to be added to conftool-data to be added to LVS endpoint and receive direct traffic.

SSL certificates

  • Elasticsearch exposes HTTPS endpoints via nginx. Puppet CA is used for certificates. Certificates have to be regenerated after the first installation to ensure they contain the correct SAN entry. The procedure is documented on Puppet CA page.

WMF Production setup

DC cluster name elastic https port elastic transport port envoy port endpoint
eqiad production-search-eqiad (chi) 9243 9300 6102 search.svc.eqiad.wmnet
eqiad production-search-omega-eqiad 9443 9500 6103 search.svc.eqiad.wmnet
eqiad production-search-psi-eqiad 9643 9700 6104 search.svc.eqiad.wmnet
codfw production-search-codfw (chi) 9243 9300 6202 search.svc.codfw.wmnet
codfw production-search-omega-codfw 9443 9500 6203 search.svc.codfw.wmnet
codfw production-search-psi-codfw 9643 9700 6204 search.svc.codfw.wmnet
eqiad cloudelastic-chi-eqiad 9243 9300 6105 cloudelastic.wikimedia.org
eqiad cloudelastic-omega-eqiad 9443 9500 6106 cloudelastic.wikimedia.org
eqiad cloudelastic-psi-eqiad 9643 9700 6107 cloudelastic.wikimedia.org
Load balancing requests

MediaWiki talks to the elasticsearch cluster through LVS. Application servers talk to a single LVS ip address and this balances the requests out across the cluster. The read and write requests are distributed evenly across all available elasticsearch instances with no consideration for data locality. The recieving elasticsearch instance turns the request into 1-N shard requests and routes the work to appropriate data nodes over the inter-node transport.

Shard balance across the cluster

The production configuration sets index.routing.allocation.total_shards_per_node to 1 for most indexes. This means that each server will only contain a single shard for any given index. This combined with setting the number of shards and replicas to an appropriate number ensure that the indices with heavy query volume are spread out across the entire cluster.

The elasticsearch shard distribution algorithm is relatively naive with respect to our use case. It is optimized for having shards of approximately equal size and query volume in all indexes. The WMF use case is very different. As of September 2015 there are 6419 shards with approximately the following size breakdown:

size num shards percent
x > 50gb 33 0.5%
50gb > x > 10gb 56 0.8%
10gb > x > 1gb 1307 20.3%
1gb > x > 100mb 1114 17.3%
100mb > x 3909 60.8%

Due to this load across the cluster needs to be occasionally monitored and shards moved around. This is further discussed in #Trouble.

Indexing

CirrusSearch updates the elasticsearch index by building and upserting almost the entire document on every edit. The revision id of the edit is used as the elasticsearch version number to ensure out-of-order writes by the job queue have no effect on the index correctness. There are a few sources of writes to the production search clusters, although CirrusSearch is the majority of writes. Writes also come from:

  • Cirrus Streaming Updater, a flink application that is due to replace the writes performed by the job queue
  • mjolnir-bulk-daemon, run on search-loader instances, pushes updates generated by teams airflow instance into the search clusters. This is primarily the weighted_tags field.
  • logstash, run on apifeatureusage instances, writes to it's own indexes in the search clusters

Streaming Updater

NOTE: Updater deployment started early 2024. Writes may be split between this and the old realtime update process

The Cirrus Streaming Updater (SUP) updates the elasticsearch indexes for each and every mediawiki edit. The updater is two applications, a producer and a consumer. These applications are run as helm releases in kubernetes. One producer runs per datacenter reading the events from various topics and generating a unified stream of updates that need to be applied to the search clusters. One consumer runs per cluster group (eqiad, codfw, cloudelastic) to write to. The producer only reads events from it's local datacenter. The consumer reads events from all producers in all datacenters.

The chain of events between a user clicking the 'save page' button and elasticsearch being updated is roughly as follows:

  • MW core approves of the edit and generates an event in the mediawiki.page-change stream.
  • The event is read by the streaming updater producer in the same datacenter, aggregated with related events, and results in a cirrussearch.update_pipeline.update event.
  • The consumer recieves the event, fetches the content of the update from the mediawiki api, batches together many updates, and performs a bulk write request to the appropriate elasticsearch cluster (each consumer writes to a cluster group of three elasticsearch clusters).

In addition to page change events, a singificant source of updates are mediawiki.cirrussearch.page_rerender events. These events represent changes that did not change the revision_id, but likely changed the rendered result of the page (ex: propagated template updates). While these are the higher volume inputs, the updater reads one more set of topics related to the CirrusSearch Weighted Tags functionality. These come in over a variety of streams and are generally metadata generated by async processes such as ML models or batch data processing.

Streaming updater backfilling

The streaming updater can also backfill existing indices for periods of time where updates, for whatever reason, were not written to the elasticsearch cluster. These updates are performed with the cirrus_reindex.py script in the cirrus-streaming-updater helmfile folder, the usage of which is described in multiple parts of the #Administration section.

The backfill uses a custom helm release. It runs a second copy of the standard consumer with specific constraints on kafka offsets and wikis to process.

Realtime updates

NOTE: This update process started to be replaced in early 2024. Writes may be split between this and the streaming updater.

The CirrusSearch extension updates the elasticsearch indexes for each and every mediawiki edit. The chain of events between a user clicking the 'save page' button and elasticsearch being updated is roughly as follows:

  • MW core approves of the edit and inserts the LinksUpdate object into DeferredUpdates
  • DeferredUpdates runs the LinksUpdate in the web request process, but after closing the connection to the user (so no extra delays).
  • When LinksUpdate completes it runs a LinksUpdateComplete hook which CirrusSearch listens for. In response to this hook CirrusSearch inserts CirrusSearch\Job\LinksUpdate for this page into the job queue (backed by Kafka in wmf prod).
  • The CirrusSearch\Job\LinksUpdate job runs CirrusSearch\Updater::updateFromTitle() to re-build the document that represents this page in elasticsearch. For each wikilink that was added or removed this inserts CirrusSearch\Job\IncomingLinkCount to the job queue.
  • The CirrusSearch\Job\IncomingLinkCount job runs CirrusSearch\Updater::updateLinkedArticles() for the title that was added or removed.

Other processes that write to elasticsearch (such as page deletion) are similar. All writes to elasticsearch are funneled through the CirrusSearch\Updater class, but this class does not directly perform writes to the elasticsearch database. This class performs all the necessary calculations and then creates the CirrusSearch\Job\ElasticaWrite job to actually make the request to elasticsearch. When the job is run it creates CirrusSearch\DataSender which transforms the job parameters into the full request and issues it. This is done so that any updates that fail (network errors, cluster maintenance, etc) can be re-inserted into the job queue and executed at a later time without having to re-do all the heavy calculations of what actually needs to change.

Batch updates from the database

NOTE: This update process started to be replaced in early 2024. Writes may be split between this and the streaming updater.

CirrusSearch indices can also be populated from the database to bootstrap brand new clusters or to backfill existing indices for periods of time where updates, for whatever reason, were not written to the elasticsearch cluster. These updates are performed with the forceSearchIndex.php maintenance script, the usage of which is described in multiple parts of the #Administration section.

Batch updates use a custom job type, the CirrusSearch\Job\MassIndex job. The main script iterates the entire page table and inserts jobs in batches of 10 titles. The MassIndex job kicks off CirrusSearch\Updater::updateFromPages() to perform the actual updates. This is the same process as CirrusSearch\Updater::updateFromTitle, updateFromTitle simply does a couple extra checks around redirect handling that is unnecessary here.

Scheduled batch updates from analytics network

Jobs are scheduled in the WMF analytics network by the search platform airflow instance to collect together various information collected there and ship it back to elasticsearch. The airflow jobs build one or more files per wiki containing elasticsearch bulk update statements, uploads them to swift, and sends a message over kafka indicating availability of new information to import. The mjolnir-msearch-daemon running on search-loader instances in the production network recieve the kafka messages, download the bulk updates from swift, and pipe them into the appropriate elasticsearch clusters. This includes information such as page popularity and ml predictions from various wmf projects (link recommendation, ores, more in the future).

Saneitizer (background repair process)

The saneitizer is a process to keep the CirrusSearch indices sane. It's primary purpose is to compare the revision_id held in cirrussearch and the primary wiki databases, to verify that cirrus pages are properly being updated. Pages that have a mismatched revision_id in cirrussearch and sent to the indexing pipeline to be reindexed.

The saneitizer has a secondary purpose of ensuring all indexed pages have been rendered from wikitext within the last few months. It accomplishes this by indexing every n'th page it visits is such a way that after n loops over the dataset all pages will have been re-indexed.

TODO fill in info on the saneitizer (leaving as stub for now)

Autocomplete indices

Autocomplete indices build daily via a systemd timer on mwmaint servers. Specifically, mediawiki_job_cirrus_build_completion_indices_eqiad.timer calls mediawiki_job_cirrus_build_completion_indices_eqiad.service which runs a bash script cirrus_build_completion_indices.sh , which in turn calls the CirrusSearch maintenance script UpdateSuggesterIndex.php .

Job queue

NOTE: The job-queue based update process started to be replaced in early 2024.

CirrusSearch uses the mediawiki job queue for all operations that write to the indices. The jobs can be roughly split into a few groups, as follows:

Primary update jobs

These are triggered by the actions of either users or adminstrators.

  • DeletePages - Removes titles from the search index when they have been deleted.
  • LinksUpdate - Updates a page after it has been edited.
  • MassIndex - Used by the forceSearchIndex.php maintenance script to distribute indexing load across the job runners.
Secondary update jobs

These are triggered by primary update jobs to update pages or indices other than the main document.

  • OtherIndex - Updates the commonswiki index with information about file uploads to all wikis to prevent showing the user duplicate uploads.
  • IncomingLinkCount - Triggers against the linked page when a link is added or removed from a page. Updates the list of links coming into a page from other pages on the same wiki. This is an expensive operation, and the live updates are disabled in wmf. Instead the incoming links counts are calculated in a batch by the incoming_links_weekly dag on the search-platform airflow instance and shipped as a batch update from the analytics network.
Backend write jobs

These are triggered by primary and secondary update jobs to represent an individual write request to the cluster. One job is inserted for every cluster to write to. In the backend the jobqueue is configured to partition the jobs by cluster into a separate queues. This partitioning ensures slowdowns indexing to one cluster do not cause similar slowdowns in the remaining clusters.

  • ElasticaWrite

Logging

CirrusSearch logs a wide variety of data. A few logs in particular are of interest:

  • The kibana logging dashboard for CirrusSearch contains all of the low-volume logging.
  • CirrusSearchRequests - A textual log line per request from mediawiki to elasticsearch plus a json payload of information. Logged via udp2log to mwlog1002.eqiad.wmnet. This is generally 1500-3000 log lines per second. Can be turned off by setting $wgCirrusSearchLogElasticRequests = false.
  • CirrusSearchRequestSet - This is a replacement for CirrusSearchRequests and is batched together at the php execution level. This simplifies the job of user analysis as they can look at the sum of what we did for a users requests, rather than the individual pieces. This is logged from mediawiki to the mediawiki_CirrusSearchRequests kafka topic. Can be turned off by setting $wgCirrusSearchSampleRequestSetLog = 0.

Administration

All of our (CirrusSearch's) scripts have been designated to run on mwmaint1002.eqiad.wmnet.

Hardware failures

Elasticsearch is robust to losing nodes. In case maintenance is required on a node (failed disk, RAM issues, whatever...), the node can be depooled and shutdown without any further action. There is no need to synchronize this shutdowns or restart with the Search Platform team (a ping to tell us it is happening is always welcomed).

  • any node can be taken down at any time
  • any node can rejoin the cluster at any time
  • up to 3 nodes from the same datacenter row can be taken down at any time
  • for more complex operations, please synchronize with the Search Platform team first

Tuning shard counts

Optimal shard count requires making a tradeoff between several few competing factors.

Quick background: Each Elasticsearch shard is actually a Lucene index which requires some amount of file descriptors/disk usage, compute, and RAM. So, a higher shard count causes more overhead, due to resource contention as well as "fixed costs".

Now, since Elasticsearch is designed to be resilient to instance failures, if a node drops out of the cluster, shards must rebalance across the remaining nodes (likewise for changes in instance count, etc). Shard rebalancing is rate-limited by network throughput, and thus excessively large shards can cause the cluster to be stuck "recovering" (rebalancing) for an unacceptable amount of time.

Thus the optimal shard size is a balancing act between overhead (which is optimized via having larger shards), and rebalancing time (which is optimized via smaller, more numerous shards). Less importantly, due to the problem of fragmentation, we also don't want a given shard to be too large a % of the available disk capacity.

Currently (01/07/2020 DD/MM/YY), in most cases we don't want shards to exceed 50GB, and ideally they wouldn't be smaller than 10GB (but note that for small indices this is unavoidable). Once our Elasticsearch cluster has 10G networking, we can increase our desired shard size, due to higher networking throughput decreasing the time required to redistribute large shards.

The final consideration is that different indices receive different levels of query volume. As of July 2020, enwiki and dewiki are hammered the hardest. This means nodes that have, for example,`enwiki_content shards assigned to them, will receive disproportionate load relative to nodes that lack those shards. As such, we want to set our total shard count in relation to the number of servers we have available. For example, a cluster with 36 servers would ideally have a total shard count that is close to a multiple of 36. We generally like to go slightly "under" to be able to weather losing nodes. In short, we want to maintain the invariant that most servers have the same number of shards for a given heavy index, with a few servers having 1 less shard so that we have headroom to lose nodes.

In conclusion: we want shards close to 50GB, but we also need our shard count set to avoid having any particularly hot servers, while making sure the shard count isn't completely even so that we can still afford to lose a few nodes. We absolutely want to avoid having a ridiculous number of extremely tiny shards, since the thrash/resource contention would grind the cluster to a halt.

Adding new wikis

All wikis have Cirrus enabled as the search engine. To add a new Cirrus wiki:

  1. Estimate the number of shards required (one, the default, is fine for new wikis).
  2. Create the search index
  3. Populate the search index

Create the index

mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki $wiki --cluster=all

That'll create the search index on all necessary clusters with all the proper configuration.

Populate the search index

mkdir -p ~/log
clusters='eqiad codfw cloudelastic'
for cluster in eqiad codfw cloudelastic; do
    mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --skipLinks --indexOnSkip --queue | tee ~/log/$wiki.$cluster.parse.log
    mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --skipParse --queue | tee ~/log/$wiki.$cluster.links.log
done

If the last line of the output of the --skipLinks line doesn't end with "jobs left on the queue" wait a few minutes before launching the second job. The job queue doesn't update its counts quickly and the job might queue everything before the counts catch up and still see the queue as empty. If this wiki is a private wiki then cloudelastic should be removed from the set of clusters. No harm if it's included, but it will (should?) throw exceptions and complain.

Health/Activity Monitoring

We have nagios checks that monitor Elasticsearch's claimed health (green, yellow, red) and ganglia monitoring on a number of attributes exposed by Elasticsearch.

Eqiad

Codfw

Expected eligible masters check and alert

There is an icinga check that checks that the expected number of eligible masters is exactly what it is. If this deviates, we should see icinga alerts. Check

We have nagios checks that monitor Elasticsearch's claimed health (green, yellow, red) and ganglia monitoring on a number of attributes exposed by Elasticsearch.

Check the actual eligible masters with:

curl -s localhost:9200/_cat/nodes?h=node.role,name | grep mdi

and then compare with puppet configuration.

If an expected eligible master node is down, we can either bring it up or promote another node to a master. Also make sure the masters are spread out across the rack.

Unassigned Shards Alerts

Elastic will explain the cluster allocation with the following API call:

 curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty

This should help you figure out why there are unassigned shards. One common cause is...

After a Cookbook Fails

Our rolling operation cookbook sets cluster-level shard allocation to primaries only, which speeds up operations. However, if the cookbook fails, this setting is never reverted. This may lead to alerts for unassigned shards (example at T324939). To re-enable allocation for all shards (as opposed to just primaries), remove the transient cluster settings as follows:

curl -H 'Content-Type: Application/json' -XPUT localhost:9200/_cluster/settings -d '{"transient":{ "*": null }}'

Note that the transient settings feature of the Elastic API is deprecated, so we should update our cookbooks accordingly (and update this documentation afterwards).

Monitoring the job queue

The job queue is monitored via the JobQueue Job Grafana dashboard. Unfortunately the UI is not able to autocomplete all available jobs, but the list of known jobs can be found in the JobClasses key of the CirrusSearch extension.json file. Of particular interest is usually cirrusSearchLinksUpdatePrioritized, cirrusSearchElasticaWrite, and cirrusSearchCheckerJob.

Monitoring the Streaming Updater

The streaming updater is primarily monitored via the Cirrus Streaming Updater grafana dashboard. This dashboard links to additional relevant dashboards, such as the Flink App dashboard.

Streaming Updater logging is found in App Logs - ECS (Kubernetes) in Logstash with the namespace filter set to cirrus-streaming-updater.

Waiting for Elasticsearch to "go green"

Elasticsearch has a built in cluster health monitor. red means there are unassigned primary shards and is bad because some requests will fail. Yellow means there are unassigned replica shards but the masters are doing just fine. This is mostly fine because technically all requests can still be served but there is less redundancy than normal and there are less shards to handle queries. Performance may be affected in the yellow state but requests won't just fail. Anyway, this is how you wait for the cluster to "go green".

until curl -s localhost:9200/_cluster/health?pretty | tee /tmp/status | grep \"status\" | grep \"green\"; do
  cat /tmp/status
  echo
  sleep 1
done

So you aren't just watching number scroll by for fun I'll explain what they all mean:

{
  "cluster_name" : "labs-search-project",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 30,
  "active_shards" : 55,
  "relocating_shards" : 0,
  "initializing_shards" : 1,
  "unassigned_shards" : 0
}
cluster_name
The name of the cluster.
status
The status we're waiting for.
timed_out
Has this requests for status timed out? I've never seen one time out.
number_of_nodes
How many nodes (data or otherwise) are in the cluster? We plan to have some master only nodes at some point so this should be three more than number_of_data_nodes
number_of_data_nodes
How many nodes that hold data are in the cluster?
active_primary_shards
How many of the primary shards are active? This should be indexes * shards. This number shouldn't change frequently because when the primary shards go off line one of the replica shards should take over immediately. It should be too fast to notice.
active_shards
How many shards are active? This should be indexes * shards * (1 + replicas). This will go down when a machine leaves the cluster. When possible those shards will be reassigned to other machines. This is possible so long as those machines don't already hold a copy of that replica.
relocating_shards
How many shards are currently relocating?
initializing_shards
How many shards are initializing? This mostly means they are being restored from a peer. This should max out at cluster.routing.allocation.node_concurrent_recoveries. We still use the default of 2.
unassigned_shards
How many shards have yet to be assigned? These are just waiting in line to become initializing_shards.

See the Trouble section for what can go wrong with this.

Replacing master-eligibles

To do this safely, update the elastic config to add the new masters WITHOUT removing the old masters. Restart each node in the cluster, then update the config to remove the original masters. Restart the cluster again to activate the new masters.

Restarting a node

Elasticsearch rebalances shards when machines join and leave the cluster. Because this can take some time we've stopped puppet from restarting Elasticsearch on config file changes. We expect to manually perform rolling restarts to pick up config changes or software updates (via apt-get), at least for the time being. There are two ways to do this: the fast way, and the safe way. At this point we prefer the fast way if the downtime is quick and the safe way if it isn't. The fast way in the Elasticsearch recommended way. The safe way keeps the cluster green the whole time but is really slow and can cause the cluster to get a tad unbalanced if it is running close to the edge on disk space.

The fast way:

es-tool restart-fast

This will instruct Elasticsearch not to allocate new replicas for the duration of the downtime. It should be faster then the simple way because unchanged indexes can be restored on the restarted machine quickly. It still takes some time.

The safe way starts with this to instruct Elasticsearch to move all shards off of this node as fast as it can without going under the appropriate number of replicas:

Banning nodes from the cluster

Via Cookbook

If you have SRE-level access, you can run the ban cookbook from a cumin host.

ip=$(facter ipaddress)
curl -H 'Content-Type: Application/json' -XPUT localhost:9200/_cluster/settings?pretty -d "{
    \"transient\"\: {
        \"cluster.routing.allocation.exclude._ip\": \"$ip\"
    }
}"

Note: Although the transient setting is deprecated and Elastic recommends using "persistent" instead, "transient" has more priority. [citation needed]

Now you wait for all the shards to move off of this one:

host=$(facter hostname)
function moving() {
   curl -s localhost:9200/_cluster/state | jq -c '.nodes as $nodes |
       .routing_table.indices[].shards[][] |
       select(.relocating_node) | {index, shard, from: $nodes[.node].name, to: $nodes[.relocating_node].name}'
}
while moving | grep $host; do echo; sleep 1; done

Now you double check that they are all off. See the advice under stuck in yellow if they aren't. It is probably because Elasticsearch can't find another place for them to go:

curl -s localhost:9200/_cluster/state?pretty | awk '
    BEGIN {more=1}
    {if (/"nodes"/) nodes=1}
    {if (/"metadata"/) nodes=0}
    {if (nodes && !/"name"/) {node_name=$1; gsub(/[",]/, "", node_name)}}
    {if (nodes && /"name"/) {name=$3; gsub(/[",]/, "", name); node_names[node_name]=name}}
    {if (/"node"/) {node=$3; gsub(/[",]/, "", node)}}
    {if (/"shard"/) {shard=$3; gsub(/[",]/, "", shard)}}
    {if (more && /"index"/) {
        index_name=$3
        gsub(/[",]/, "", index_name)
        print "node="node_names[node]" shard="index_name":"shard
    }}
       ' | grep $host

Now you can restart this node and the cluster with stay green:

sudo /etc/init.d/elasticsearch restart
until curl -s localhost:9200/_cluster/health?pretty ; do
    sleep 1
done

Next you tell Elasticsearch that it can move shards back to this node:

curl -H 'Content-Type: Application/json' -XPUT localhost:9200/_cluster/settings?pretty -d "{
    \"transient\"": {
        \"cluster.routing.allocation.exclude._ip\"\: \"\"
    }
}"

Finally you can use the same code that you used to find stuff shards moving from this node to find shards moving to this node:

while moving | grep $host; do echo; sleep 1; done

Rolling restarts

Rolling restarts are done with multiple servers in parallel to speed up the operation. Servers from the same availability zone (datacenter row) are restarted at the same time to ensure that all indices still have sufficient shards. 3 nodes in parallel is conservative, 4 has been tested multiple times and works if the cluster isn't under high load.

To optimize recovery, writes are paused during restart and a sync flush is issued. Pausing writes will increase backlog in the kafka queues and writes will be dropped if writes are paused for too long (at this point ~3 hours). Writes are re-enabled between each group of restarts to allow the queue to be processed.

Each server runs multiple elasticsearch instances (2 at this point), both need to be restarted.

The full restart logic is embedded in cookbooks for the different specific use cases:

Those cookbooks are well tested on the main elasticsearch clusters, but not as much on the smaller ones (relforge, cloudelastic). The small clusters are simple enough to restart manually. Those cookbooks are idempotent, they can be stopped and restarted, servers already restarted will be ignored by the next run. To be on the safe side, stop the cookbook while they are waiting on the cluster to go back to green (context managers are used for all operations that needs to be undone, but a force kill might not leave them time to cleanup).

Example run: On one of the cumin host:

sudo -i cookbook sre.elasticsearch.rolling-restart search_eqiad "restart for JVM upgrade" --start-datetime 2019-06-12T08:00:00 --nodes-per-run 3

where:

  • search_eqiad is the cluster to be restarted
  • --start-datetime 2019-06-12T08:00:00 is the time at which the operation is starting (which allows the cookbook to be restarted without restarting the already restarted servers).
  • --nodes-per-run 3 is the maximum number of nodes to restart concurrently

During rolling restarts, it is a good idea to monitor a few elasticsearch specific things:

Things that can go wrong:

  • some shards are not reallocated: Elasticsearch stops trying to recover shards after too many failures. To force reallocation, use the sre.elasticsearch.force-shard-allocation cookbook.
  • writes are not thawed: in some rare cases (that we don't entirely understand) writes are frozen during the restart, but not thawed properly. Use the sre.elasticsearch.force-unfreeze cookbooks to manually recover.
  • master re-election takes too long: There is no way to preemptively force a master relection. When the current master is restarted, an election will occur. This sometimes takes long enough that it has impact. This might raise an alert and some traffic might be dropped. This recovers as soon as the new master is elected (1 or 2 minutes). We don't have a good way around this at the moment.
  • cookbook are force killed or in error: The cookbook use context manager for most operations that need to be undone (stop puppet, freeze writes, etc...). A force kill might not leave time to cleanup. Some operations are not rolled back in case of exception, like pool / depool, because unknown exception might leave the server in an unkown state and do require manual checking.

Cold restart

A cold restart (shutting down all nodes and restarting all of them at the same time) is governed by the following settings:

  • recover_after_nodes
  • recover_after_time
  • expected_nodes

Our current understanding is that elasticsearch will wait for expected_nodes to be available, and recover anyway after recover_after_time if at least recover_after_nodes are present.

Recovering from an Elasticsearch outage/interruption in updates

Primary shard lost, replica shard considered outdated

This can happen on our smaller clusters (cloudelastic or relforge) as they have little to no redundancy. You can tell Elastic to use a shard it considers outdated with the following call. After this the backfill procedure should be run over the time period the promoted replica missed:

curl -XPOST -H 'Content-Type: application/json' https:// cloudelastic.wikimedia.org:9243/_cluster/reroute -d'
{
   "commands": [
       {"allocate_stale_primary": {
           "index": "wikidatawiki_content_1692131793",
           "shard": 19,
           "accept_data_loss": true,
           "node": "cloudelastic1003-cloudelastic-chi-eqiad"
       }}
   ]}'
   

Backfilling

The same script that populates the search index can be run over a more limited list of pages:

user@mwmaint:~$mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki $wiki --from <YYYY-mm-ddTHH:mm:ssZ> --to <YYYY-mm-ddTHH:mm:ssZ>

or:

user@mwmaint:~$mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki $wiki --fromId <id> --toId <id>

So the simplest way to recover from an Elasticsearch outage should be to use --from and --to with the times of the outage. Don't be stingy with the dates - it is better to reindex too many pages than too few.

If there was an outage its probably good to also do:

user@mwmaint:~$ mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki $wiki --deletes --from <YYYY-mm-ddTHH:mm:ssZ> --to <YYYY-mm-ddTHH:mm:ssZ>

This will pick up deletes which need to be iterated separately.

This is the script I have in my notes for recovering from an outage:

function outage() {
   wiki=$1
   start=$2
   end=$3
   mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --from $start --to $end --deletes | tee -a ~/cirrus_log/$wiki.outage.log
   mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --from $start --to $end --queue | tee -a ~/cirrus_log/$wiki.outage.log
}
while read wiki ; do
   outage $wiki '2015-03-01T20:00:00Z' '2015-03-02T00:00:00Z'
done < /srv/mediawiki/dblists/all.dblist

Streaming Updater backfill

In wikis that have transitioned to the streaming updater the backfill procedure is wrapped up into a python script. This script is found in the cirrus-streaming-updater directory of the deployment-charts repo on the deployment hosts. Backfill all configured wikis for the time period:

cd /srv/deployment-charts/helmfile.d/services/cirrus-streaming-updater
python3 ./cirrus_reindex.py eqiad backfill <YYYY-mm-ddTHH:mm:ss> <YYYY-mm-ddTHH:mm:ss>

Backfill specific wikis for a given time period:

python3 ./cirrus_reindex.py eqiad <wiki1> <wiki2> <...> backfill <YYYY-mm-ddTHH:mm:ss> <YYYY-mm-ddTHH:mm:ss>

In-place reindexing is similar, but uses different arguments. This will invoke mwscript to perform the reindexing, and then backfill the time period that the reindexing was running. This is limited to operating on a single wiki at a time:

python3 ./cirrus_reindex.py eqiad testwiki reindex

In-place reindexing all wikis then looks like:

 cluster=eqiad
 cd /srv/deployment-charts/helmfile.d/services/cirrus-streaming-updater
 expanddblist all | while read wiki ; do
     python3 ./cirrus_reindex.py $cluster $wiki reindex | tee -a $HOME/cirrus_log/$wiki.$cluster.reindex.log
 done

In place reindex

Some releases require an in-place reindex. Usually in-place reindex is needed if any part of mapping: fields, analyzers, etc. - has changed. This mode of reindexing creates a new index with up-to-date mappings, but does not change the source data already stored in the index, only the interpretation of the data. All reindex operations are very expensive, it can take multiple weeks to sequentially work through the wikis and clusters.

Sometime the change may be only for wikis in particular languages so only those wikis will need an update. In any case, this is how you perform an in-place reindex:

function reindex() {
    cluster=$1
    wiki=$2
    mkdir -p "$HOME/cirrus_log/"
    reindex_log="$HOME/cirrus_log/$wiki.$cluster.reindex.log"     
    if [ -z "$cluster" -o -z "$wiki" ]; then
        echo "Usage: reindex [cluster] [wiki]"
        return 1
    fi
    TZ=UTC export REINDEX_START=$(date +%Y-%m-%dT%H:%M:%SZ)
    echo "Started at $REINDEX_START" > $reindex_log
    mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki $wiki --cluster $cluster --reindexAndRemoveOk --indexIdentifier now 2>&1 | tee -a $reindex_log && \
    mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --deletes | tee -a $reindex_log && \
    mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --queue   | tee -a $reindex_log
    mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --archive | tee -a $reindex_log
}

If you only need to reindex a certain elasticsearch index/wiki type (for example, just dewiki_content rather than all of dewiki, use the following altered version of the reindex function:

function reindex_single_es_index() {
   cluster="$1"
   wiki="$2"
   es_index_suffix="$3"
   mkdir -p "$HOME/cirrus_log/"
   reindex_log="$HOME/cirrus_log/$wiki.$cluster.reindex.log"
   if [ -z "$cluster" -o -z "$wiki" -o -z "es_index_suffix" ]; then
       echo "Usage: reindex [cluster] [wiki] [es_index_suffix]"
       return 1
   fi
   TZ=UTC export REINDEX_START=$(date +%Y-%m-%dT%H:%M:%SZ)
   echo "Started at $REINDEX_START" > "$reindex_log"
   mwscript extensions/CirrusSearch/maintenance/UpdateOneSearchIndexConfig.php --wiki $wiki --cluster $cluster --indexType $es_index_suffix --reindexAndRemoveOk --indexIdentifier now 2>&1 | tee -a "$reindex_log" && \
   mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --deletes | tee -a "$reindex_log" && \
   mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --queue   | tee -a "$reindex_log"
   mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --archive | tee -a "$reindex_log"
}


Every cluster is isolated, so it is suggested to run a reindex process per-cluster. Essentially create a tmux session with a pane for each cluster, and run the following in each (with CLUSTER adjusted as appropriate). This must be done for eqiad, codfw and cloudelastic clusters.

Because the CirrusSearch reindex process involves hitting the SQL databases (data is fed from SQL into Elasticsearch), it's helpful to have each tmux pane run the reindex in a slightly different order to avoid thrash. (This is a recommended performance optimization but is not mandatory, to be clear)

cluster=eqiad
expanddblist all | while read wiki ; do
    reindex $cluster $wiki
done


Don't worry about incompatibility during the update - CirrusSearch is maintained so that queries and updates will always work against the currently deployed index as well as the new index configuration. Once the new index has finished building (the second command) it'll replace the old one automatically without any interruption of service. Some updates will have been lost during the reindex process. The third command will catch those updates.

Full reindex

If for some reason a wiki index has to be recreated from scratch (bad backups, no other cluster to copy from, some other sort of total failure) CirrusSearch can regenerate the index from the SQL databases. Historically this was also used to add new fields or apply changes to how cirrussearch renders fields, but that use case has been replaced by the Saneitizer background process. This process is expensive and will take significant time, it is a last resort.

First, do an in-place reindex as above. This is necessary due to the fact that in-place indexing updates the index mappings, while full reindex uses existing mappings to bring in the new data. Then, use this to make scripts that run the full reindex:

 function make_index_commands() {
     wiki=$1
     mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki $wiki --queue --maxJobs 10000 --pauseForJobs 1000 \
         --buildChunks 250000 |
         sed -e 's/^php\s*/mwscript extensions\/CirrusSearch\/maintenance\//' |
         sed -e 's/$/ | tee -a cirrus_log\/'$wiki'.parse.log/'
 }
 function make_many_reindex_commands() {
     wikis=$(pwd)/$1
     rm -rf cirrus_scripts
     mkdir cirrus_scripts
     pushd cirrus_scripts
     while read wiki ; do
         make_index_commands $wiki
     done < $wikis | split -n r/5
     for script in x*; do sort -R $script > $script.sh && rm $script; done
     popd
 }

Then run the scripts it makes in screen sessions.

Cluster Quorum Loss Recovery Procedure

If the cluster loses quorum, Elastic will shut down external access. elasticsearch-node , which is included in the elasticsearch Debian package, can recreate a cluster with its last-known state, but it carries the risk of data loss. Only use if there are no other options.

  1. Stop Elasticsearch service everywhere (example command: systemctl stop elasticsearch_7@production-search-omega-codfw.service)
  2. Run elasticsearch-node on all master-eligibles, taking care to target the correct Elastic cluster (to do this, set the ES_PATH_CONF env var). DO NOT CONFIRM THE COMMAND YET! Example command: export ES_PATH_CONF=/etc/elasticsearch/production-search-omega-codfw; elasticsearch-node unsafe-bootstrap
  3. Check the output of elasticsearch-node; it contains a value that represents the cluster state. Example output: Current node cluster state (term, version) pair is (4, 12). Find the host with the highest number, that represents the newest cluster state and that will be your initial master on the new cluster you are forming.
  4. Run export ES_PATH_CONF=/etc/elasticsearch/production-search-omega-codfw; elasticsearch-node unsafe-bootstrap on the initial master and confirm. Next, start the elasticsearch service: systemctl start elasticsearch_7@production-search-omega-codfw.service). You now have a new cluster! Now we need to get the other nodes to forget about their old cluster.
  5. On the remaining master-eligibles, run export ES_PATH_CONF=/etc/elasticsearch/production-search-omega-codfw; elasticsearch-node detach-cluster and confirm. This wipes the old cluster state clean. Next, start the Elastic service on all remaining master-eligibles: systemctl start elasticsearch_7@production-search-omega-codfw.service
  6. Confirm that the other master-eligibles have joined the cluster. Example command: curl http://0:9400/_cat/nodes
  7. Repeat step 5 on all data nodes.
  8. Confirm that the cluster is healthy again: curl http://0:9400/_cat/health?v=true Note that the data recovery will take some time; you might still get a 503 for several minutes after bringing all nodes into the cluster.

Dumps

Dumps of all indices are created weekly by a cron as part of the snapshot puppet module. The dumps are available on dumps.wikimedia.org and can be reimported via the elasticsearch _bulk API.

Adding new masters/removing old masters

  1. Add new masters as "unicast_hosts" in our production puppet repo's cirrus.yaml file.
  2. After/merging/puppet-merging, run the rolling operation cookbook with the "restart" argument, this brings the new masters into the cluster.
  3. Check the master status in the elastic API. For example: curl -s -XGET "https://search.svc.codfw.wmnet:9243/_cat/nodes". The current master has an asterisk in the second-to-last field. Example output: 10.192.16.229 61  98 10 4.62 5.14 5.07 dimr * elastic2093-production-search-codfw
  4. If one of the old hosts is still master, force a failover. Do this carefully (one host at a time) until one of the new master-eligibles is the active master.
  5. Remove old hosts from "unicast_hosts" in our production puppet repo's cirrus.yaml file, but DO NOT remove them from the cluster completely (yet).
  6. After/merging/puppet-merging, run the rolling operation cookbook with the "restart" argument, again. The old master eligibles will lose their master-eligible status after a cluster restart, which is what we want.
  7. Make/merge a new puppet patch completely removing the old master-eligibles from the cluster.

After this change is merged (and puppet-merged), we also need to run this script, which informs the other ES clusters of the new masters. The script requires manually creating an 'lst' file, here's an example on how to do that.

Adding new nodes

  • (Step 1a) If they're cirrus (elasticsearch*) hosts, add the racking info for the new hosts in hieradata/regex.yaml (you can use Netbox to easily get the rack/row information, by searching like so). Otherwise for cloudelastic, add the entries in a yaml file per host in hieradata/hosts.
  • (Step 1b) Also add the new nodes to the cluster_hosts list for each of the 3 elasticsearch instances in hieradata/role/$DATA_CENTER/elasticsearch/cirrus.yaml (example patch). Put every host in the main cluster, but divide the hosts between the two smaller clusters as evenly as possible (recall that each cirrus elastic* host runs two instances of Elasticsearch). Puppet will need to run on all existing elastic* hosts in the same datacenter in order for them to establish their LVS rules that allow communication with the new hosts.
  • (Step 1d) Add new entries in conftool-data/node/$DATACENTER.yaml. These entries need to specify which of the two smaller clusters (psi or omega) a given host belongs to. (example patch)
  • (Step 2) Flip the not-yet-in-service insetup hosts to role::elasticsearch::cirrus in manifests/site.pp (example patch). Run puppet on the new hosts twice (currently this needs to be done twice due to nonideal order of operations in our/o11y's puppet logic)
  • (Step 3) Once the hosts have joined the clusters and are ready to receive requests from mediawiki, we'll need to pool them and set their weights. On the puppetmaster, configure the new hosts like so: sudo confctl select name=elastic206[1-9].* set/weight=10:pooled=yes. See Conftool for further documentation.

You can watch the node(s) suck up shards like so:

curl -s localhost:9200/_cluster/state?pretty | awk '
    BEGIN {more=1}
    {if (/"nodes"/) nodes=1}
    {if (/"metadata"/) nodes=0}
    {if (nodes && !/"name"/) {node_name=$1; gsub(/[",]/, "", node_name)}}
    {if (nodes && /"name"/) {name=$3; gsub(/[",]/, "", name); node_names[node_name]=name}}
    {if (/"RELOCATING"/) relocating=1}
    {if (/"routing_nodes"/) more=0}
    {if (/"node"/) {from_node=$3; gsub(/[",]/, "", from_node)}}
    {if (/"relocating_node"/) {to_node=$3; gsub(/[",]/, "", to_node)}}
    {if (/"shard"/) {shard=$3; gsub(/[",]/, "", shard)}}
    {if (more && relocating && /"index"/) {
        index_name=$3
        gsub(/[",]/, "", index_name)
        print "from="node_names[from_node]" to="node_names[to_node]" shard="index_name":"shard
        relocating=0
    }}
       '

Removing nodes

Push all the shards off the nodes you want to remove like this:

curl -H 'Content-Type: Application/json' -XPUT localhost:9200/_cluster/settings -d '{
    "transient"": {
        "cluster.routing.allocation.exclude._ip"p: "10.x.x.x,10.x.x.y,..."
    }
}'

Then wait for there to be no more relocating shards:

echo "Waiting until relocating shards is 0..."
until curl -s localhost:9200/_cluster/health?pretty | tee /tmp/status | grep '"relocating_shards" : 0,'; do
   cat /tmp/status | grep relocating_shards
   sleep 2
done

Use the cluster state API to make sure all the shards are off of the node:

curl localhost:9200/_cluster/state?pretty | less

Elasticsearch will leave the shard on a node even if it can't find another place for it to go. If you see that then you have probably removed too many nodes. Finally, you can decommission those nodes in puppet. The is no need to do it one at a time because those nodes aren't hosting any shards any more.

Standard decommissioning documentation applies. When decommissioning nodes, you should ensure that elasticsearch is cleanly stopped before the node is cut from the network.

Deploying Debian Packages

When we upgrade our Elasticsearch or Debian distro versions, we have to redeploy packages, including:

  • elasticsearch-madvise
  • liblogstash-gelf-java
  • wmf-elasticsearch-search-plugins

In the case where upstream has not changed significantly, you (an SRE or someone with SRE access) can simply copy the Debian package from the previous distro repository, as described on the reprepro page.

Deploying plugins

Plugins (for search) are deployed using the wmf-elasticsearch-search-plugins debian package. This package is built using the ops-es-plugins repository and deployed to our apt repo like any other debian packages we maintain.

The list of plugins we currently maintain is:

They are currently released using the classic maven process (release:prepare|perform) and deployed to maven central. The repository name used for deploying to central is ossrh and thus the engineer performing the release must have its ~/.m2/settings.xml with the following credentials set:

<settings>
    <servers>
        <server>
            <id>ossrh</id>
            <username>username</username>
            <password>password</password>
        </server>
    </servers>
</settings>

To request write permission to this repository one must create a request like this one and obtain a +1 from a person already granted.

Please see discovery-parent-pom for generic advises on the build process of the java projects we maintain.

Backup and Restore

Via Elastic S3 Snapshot Repository Plugin

See this page for more details.

Inspecting logs

There is two different types of log, we have elasticsearch logs on each node and cirrus logs.

Elasticsearch logs

The logs generated by elasticsearch are located in /var/logs/elasticsearch/ :

  • production-search-eqiad.log is the main log, it contains all errors (mostly failed queries due to syntax errors). It's generally a good idea to keep this log opened on the master and the node involved in administration tasks.
  • production-search-eqiad_index_search_slowlog.log contains the queries that are considered slow (thresholds are described in /etc/elasticsearch/elasticsearch.yml)
  • elasticsearch_hot_threads.log is a snapshot of java threads considered "hot" (generated by the python script elasticsearch_hot_threads_logger.py, it takes a snapshot from hot threads every 10 or 50 seconds depending on the load of the node)

NOTE: the log verbosity can be changed at runtime, see elastic search logging for more details.

Cirrus logs

The logs generated by cirrus are located on mwlog1001.eqiad.wmnet:/a/mw-log/ :

  • CirrusSearch.log: the main log. Around 300-500 lines generated per second.
  • CirrusSearchRequests.log: contains all requests (queries and updates) sent by cirrus to elastic.Generates between 1500 and 3000+ lines per second.
  • CirrusSearchSlowRequests.log: contains all slow requests (the threshold is currently set to 10s but can be changed with $wgCirrusSearchSlowSearch). Few lines per day.
  • CirrusSearchChangeFailed.log: contains all failed updates. Few lines per day except in case of cluster outage.

Useful commands :

See all errors in realtime (useful when doing maintenance on the elastic cluster)

tail -f /a/mw-log/CirrusSearch.log | grep -v DEBUG

WARNING: you can rapidly get flooded if the pool counter is full.

Measure the throughput between cirrus and elastic (requests/sec) in realtime

tail -f /a/mw-log/CirrusSearchRequests.log | pv -l -i 5 > /dev/null

NOTE: this is an estimation because I'm not sure that all requests are logged here. For instance: I think that the requests sent to the frozen_index are not logged here. You can add something like 150 or 300 qps (guessed by counting the number of "Allowed write" in CirrusSearch.log)

Measure the number of prefix queries per second for enwiki in realtime

tail -f /a/mw-log/CirrusSearchRequests.log | grep enwiki_content | grep " prefix search " | pv -l -i 5 > /dev/null

Using jstack or jmap or other similar tools to view logs

Our elasticsearch systemd unit sets PrivateTmp=true, which is overall a good thing. But this prevents jstack / jmap / etc. from connecting to the JVM as they expect a socket in the temp directory. The easy/safe workaround to viewing these logs is via nsenter (See T230774). Example:

gehel@elastic2050:~$ sudo systemctl status elasticsearch_6@production-search-codfw.service | grep "Main PID"
 Main PID: 1110 (java)
gehel@elastic2050:~$ sudo nsenter -t 1110 -m sudo -u elasticsearch jstack 1110
2019-08-20 08:58:53
Full thread dump OpenJDK 64-Bit Server VM (25.222-b10 mixed mode):

"Attach Listener" #9275 daemon prio=9 os_prio=0 tid=0x00007f9c48002000 nid=0x31f81 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"elasticsearch[elastic2050-production-search-codfw][fetch_shard_store][T#15]" #7796 daemon prio=5 os_prio=0 tid=0x00007f9b58025000 nid=0x2a438 waiting on condition [0x00007f8a64a5e000]
   java.lang.Thread.State: WAITING (parking)
[...]
or We can go further with:
systemctl-jstack() {
if [ -z $1 ] {
   echo "Please provide a service name."
   echo "Usage: systemctl-jstack  elasticsearch_6@production-search-codfw.service"
   exit 1
}
PID=$(systemctl show $1  -p MainPID | sed -e 's/^MainPID=//')
USER=$(systemctl show $1 -p User | sed - e 's/^User=//')
sudo nsenter -t $PID -m sudo -u $USER jstack $PID
}

Multi-DC / Multi-Cluster Operations

We have deployed full size elasticsearch clusters in eqiad and codfw. The cluster are queried via LVS at search.svc.{eqiad,codfw}.wmnet.

Read Operations

By default application servers are configured to query the elasticsearch cluster in their own datacenter. Traffic can be shifted to a single datacenter for maintenance operations when necessary via the $wgCirrusSearchDefaultCluster and $wgCirrusSearchClusterOverrides configuration in mediawiki-config.

Write Operations

Elasticsearch does not have support for replicating search indices across clusters, instead we perform all writes to all clusters we want to store the data.

In the classic update pipeline this is done via the job queue by enqueueing an ElasticaWrite job per cluster that needs to be written to. These jobs are partitioned by cpjobqueue such that the write jobs are consumed independantly, a slowdown on one cluster indexing does not effect the others. Writes are performed by the job queue in the active mediawiki datacenter to all clusters in all datacenters.

In the replacement streaming updater pipeline a producer application in each datacenter reads events from mediawiki in the same datacenter and generates a stream of pages that need to be updated. That stream of updates is mirrored to all datacenters by the kafka infrastructure. A consumer application running in the same datacenter as the elasticsearch cluster consumes the updates from all datacenters and applies them to the local cluster.

DC Switch

If wgCirrusSearchDefaultCluster is set to $GLOBALS['wmgDatacenter'] in ext-CirrusSearch.php everything should be automatic. If a specific DC is pinned, please do coordinate with the Search Team to understand why (some elasticsearch maintenance is perhaps in progress).

Removing Duplicate Indices

When index creation bails out, perhaps due to a thrown exception or some such, the cluster can be left with multiple indices of a particular type but only one active. CirrusSearch contains a script, scripts/check_indices.py, that will check the configuration of all wikis in the cluster and then compare the set of expected indices with those actually in the elasticsearch clusters.


/srv/mediawiki/php/extensions/CirrusSearch/scripts/check_indices.py | jq .

When everything is expected the output will be an empty json array. Note that it might take a minute or three before giving an answer:

 []

When something is wrong the output will contain an object for each cluster, and then the cluster will have a list of problems found:

 [
   {
     "cluster_name": "production-search-eqiad",
     "problems": [
       {
         "reason": "Looks like Cirrus, but did not expect to exist.Deleted wiki?",
         "index": "ebernhardson_test_first"
       }
     ],
     "url": "https://search.svc.eqiad.wmnet:9243"
   }
 ]

Indicies still have to be manually deleted after reviewing the output above. As we gain confidence in the output through manual review the tool can be updated with options to automatically apply deletes.

curl -XDELETE https://search.svc.eqiad.wmnet/ebernhardson_test_first

Rebalancing shards to even out disk space use

Viewing free disk space per node:

curl -s localhost:9200/_cat/nodes?h=d,h | sort -nr

To move shards away from the nodes with the least disk free, we can lower cluster.routing.allocation.disk.watermark.high settings temporarily. The high watermark is the limit at which Elasticsearch will start moving shards out of the node to free up space. High watermark can be modified by:

curl -H 'Content-Type: Application/json' -XPUT localhost:9200/_cluster/settings -d '{
    "transient"": {
        "cluster.routing.allocation.disk.watermark.high"h: "70%"
    }
}'

Keep an eye on the number of indices on a few nodes

When banning nodes to prepare for some maintenance operations, it is useful to see how many shards are left on those nodes:

 watch -d -n 10 \
 "curl -s localhost:9200/_cat/shards \
   | awk '{print \$8}' \
   | egrep 'elastic10(21|22|23|24)' \
   | sort \
   | uniq -c"

Elasticsearch Curator

elasticsearch-curator is a python tool which provides high level configuration for some maintenance operations. Its configuration is based on action files. Some standard actions are deployed on each elasticsearch node in /etc/curator. For example, you can disable shard routing with:

 sudo curator --config /etc/curator/config.yaml /etc/curator/disable-shard-allocation.yaml

Curator must be run as root to access its log file (/var/log/curator.log).

Explain unallocated shards

Elasticsearch 5.2 introduced a new API endpoint to explain why a shard is unassigned:

 curl -s localhost:9200/_cluster/allocation/explain?pretty
Anti-affinity (shards limit) prevents shards from assignment

Our Elastic hosts are spread across four rows in the datacenter (A-D). Within a single row, we allow no more than 2 replicas per shard. This can lead to unassigned replicas, especially during maintenance operations. Check for this with the following command:

curl -s -X GET "localhost:9200/_cluster/allocation/explain?pretty" -H 'Content-Type: application/json' -d'
{ "index": "enwiki_content_1658309446", "shard": 15,
"primary": false
} ' | jq '.node_allocation_decisions[] | select (.node_attributes.row == "C")'

Shards stuck in recovery

We've seen cases where some shards are stuck in recovery and never complete the recovery process. Reducing temporarily the number of replicas and increasing it again once the cluster is green seems to fix the issue. So far, this has happened only during the 5.6.4 to 6.5.4 upgrade.

   curl -k -X PUT "https://localhost:9243/$index/_settings" -H 'Content-Type: application/json' -d'
   {
       "index" : {
           "auto_expand_replicas" : "0-2",
           "number_of_replicas" : 2
       }
   }
   '

No updates

If no updates are flowing, the usual culprit can be:

  • Kafka queues are having issues (or eventgate, writing to kafka for mediawiki).
    • These should be causing alerts in #wikimedia-operations
  • Job runner are having issues
    • Review JobQueue Job dashboard to look for backlogging
  • Streaming Updater consumers or producers having issues
    • Review Streaming Updater and Flink App grafana dashboards
    • Check on pod lifetimes. If they are single digit minutes the problem could be there.

Stuck in old GC hell

When memory pressure is excessive the GC behaviour of an instance will switch from primarily invoking the young GC to primarily invoking the old GC. If the issue is limited to a single instance it's possible the instance has an unlucky set of shards that require more memory than the average instance. Resolving this situation involves:

  1. Ban the node from cluster routing
  2. Wait for all shards to drain
  3. Restart instance
  4. Unban the node from cluster routing

Draining all the shards away ensures the instance will get a new selection of shards when it is unbanned. Restarting the instance is not strictly required, but it's nice to start with a fresh JVM after the old one has been misbehaving.

Pool Counter rejections (search is currently too busy)

Sometimes users' searches may fail with the message: "An error has occurred while searching: Search is currently too busy. Please try again later."

This is generally due to a spike in traffic which triggers MediaWiki's PoolCounter, which in the case of CirrusSearch will drop requests if the traffic spike is too large.

There is an alert mediawiki_cirrus_pool_counter_rejections_rate which should warn if a concerning number of CirrusSearch PoolCounter rejections are detected. The alert can be checked in icinga.

If the alert fires, take some time to consider if the current PoolCounter thresholds should be increased to increase the allowable queue size on MediaWiki's end. If the threshold cannot be increased, then all there is to do is verify that the traffic spike is organic as opposed to the result of some bug upstream.

Indexing hung and not making progress

Elastic nodes can get hung up indexing and stop doing work. We've seen this happen from a bug in our code, although theoretically it could be caused by other issues related to ingest (bad data, too much data, etc). This manifests as increased Search Thread Pool Rejections per Second. When this happens, you may have to temporarily shut off mjolnir services on search-loader hosts to alleviate backpressure.

Deploying Elasticsearch config (.yml/jvm) changes

Elasticsearch needs to be restarted in a controlled manner. When puppet deploys configuration changes for elasticsearch the changes are applied on disk but the running service is not restarted. After puppet has succesfully run on all instances follow the #Rolling restarts runbook.

CirrusSearch titlesuggest index is too old

This alert is fired when titlesuggest indices, which power fuzzy title complete, are older than expected for at least one wiki. This is not an urgent problem, the old version of these indices covers most use cases. Over time the changes compound and become more noticable. The exact indices that are having problems can be checked with the following. This will give the 5 oldest titlesuggest indices for the given cluster.

curl -XPOST -H 'Content-Type: application/json' 'https://search.svc.eqiad.wmnet:9243/*_titlesuggest,omega:*_titlesuggest,psi:*_titlesuggest/_search' -d '{
    "size": 0,
    "aggs": {
        "by_index": {
            "terms": {
                "field": "_index",
                "order": {"max_batch_id": "asc"}
            },
            "aggs": {
                "max_batch_id": {"max":{"field": "batch_id"}}
            }
        }
    }
}
' | jq '.aggregations.by_index.buckets[].max_batch_id.value |= (. | strftime("%Y-%m-%d %H:%M UTC"))'

The titlesuggest indices are built by the mediawiki_job_cirrus_build_completion_indices_{eqiad,codfw} systemd timer typically run on mwmaint1002.eqiad.wmnet. The relevant logs, in mwmaint1002.eqiad.wmnet:/var/log/mediawiki/mediawiki_job_cirrus_build_completion_indices_{eqiad,codfw} should be checked to find out why the given titlesuggest indices are not being built.

Other elasticsearch clients

Cirrus is not the only application using our "search" elasticsearch cluster. In particular:

  • API Feature requests
  • Translate Extension
  • Phabricator
  • Toolhub

Constraints on elasticsearch clients

Reading this page should already give you some idea of how our elasticsearch cluster is managed. A short checklist that can help you:

  • Writes need to be robust: Our elasticsearch cluster can go in read-only mode for various reasons (loss of a node, maintenance operations, ...). While this is relatively rare, it is part of normal operation. The client application is responsible to implement queuing / retry.
  • Multi-DC aware: We have 2 independent clusters, writes have to be duplicated to both clusters, including index creation, the client application has to provide a mechanism to switch from one cluster to the other (in case of loss of datacenter, major maintenance on the cluster, ...)

Trouble

If Elasticsearch is in trouble, it can show itself in many ways. Searches could fail on the wiki, job queue could get backed up with updates, pool counter overflowing with unperformed searches, or just plain old high-load that won't go away. Luckily, Elasticsearch is very good at recovering from failure so most of the time these sorts of problems aren't life threatening. For some more specific problems and their mitigation techniques, see the /Trouble subpage.

Tasks Api

Sometimes pathological queries can get "stuck" burning cpu as they try and execute for many minutes. We can ask elasticsearch what tasks are currently running with the tasks api:

   curl https://search.svc.eqiad.wmnet:9243/_tasks?detailed > tasks

This will dump a multi-megabyte json file that indicates all tasks (such as shard queries, or the parent query that issued shard queries) and how long they have been running. The inclusion of the detailed flag means all shard queries will report the index they are running against, and the parent query will report the exact text of the json search query that cirrussearch provided. For an extended incident this can be useful to track down queries that have been running extended periods of time, but for brief blips it may not have much information.

For a quick look at tasks on a node, you can use something like:

   curl -s 'localhost:9200/_tasks/?detailed&nodes=_local' | \
       jq '.nodes | to_entries | .[0].value.tasks | to_entries
           | map({
               action: .value.action,
               desc: .value.description[0:100], 
               ms:(.value.running_time_in_nanos / 1000000)
           }) | sort_by(.ms)' | \
       less

Other Resources

  • The elasticsearch cat APIs. This contains much of the information you want to know about the cluster. The allocation, shards, nodes, indices, health and recovery reports within are often useful for diagnosing information.
  • The elasticsearch cluster settings api. The contains other interesting information about the current configuration of the cluster. Temporary settings changes, such as changing logging levels, are applied here.

Information about specific CirrusSearch extra functionality: