User:Giuseppe Lavagetto/Switch Dc 2017

From Wikitech

Per-service switchover instructions

MediaWiki-related

We divide the process in logical phases that should be executed sequentially. Within any phase, top-level tasks can be executed in parallel to each other, while subtasks are to be executed sequentially to each other. The phase number is referred to in the names of the tasks in operations/switchdc [1]

Phase 0 - preparation

  1. (days in advance) Warm up databases; see MariaDB/buffer_pool_dump.
  2. (days in advance) Prepare puppet patches:
    • Switch mw_primary [2]
    • Add direct route to $new_site for all mw-related cache::app_directors [3]
    • Comment-out direct route to $old_site in all mw-related cache::app_directors [4]
  3. Disable puppet on all jobqueues/videoscalers and maintenance hosts, and the varnishes
  4. Merge the mediawiki-config switchover changes but don't sync This is not covered by the switchdc script
  5. Reduce the TTL on appservers-rw, api-rw, imagescaler-rw to 10 seconds

Phase 1 - stop maintenance

  1. Stop jobqueues in the active site
  2. Kill all the cronjobs on the maintenance host in the active site

Phase 2 - read-only mode

  1. Go to read-only mode by syncing wmf-config/db-$old-site.php

Phase 3 - lock down database masters

  1. Put old-site core DB masters (shards: s1-s7, x1, es2-es3) in read-only mode.
  2. Wait for the new site's databases to catch up replication

Phase 4.1 - Wipe caches

  1. Wipe new site's memcached to prevent stale values — only once the new site's read-only master/slaves are caught up.
  2. Restart all HHVM servers in the new site to clear the APC cache

Phase 4.2 - Warmup caches in the new site

This phase will be executed by the t04_cache_wipe task of switchdc, because there is no speed gain from not doing all of phase 4.1 + phase 4.2 separately, and they are logically related.

  1. Warm up memcached and APC running the mediawiki-cache-warmup on the new site clusters, specifically:
    • The global warmup against the appservers cluster
    • The apc-warmup against all hosts in the appservers and api clusters at least.

Phase 5 - switch active datacenter configuration

  1. Merge the switch of $mw_primary at this point and add direct route to $new_site for all mw-related cache::app_directors. both changes can be puppet-merged toghether. This is not covered by the switchdc script. (Puppet is only involved in managing traffic, db alerts, and the jobrunners).
  2. Switch the discovery
    • Flip appservers-rw, api-rw, imagescaler-rw to pooled=true in the new site. This will not actually change the DNS records, but the on-disk redis config will change.
    • Deploy wmf-config/ConfigSettings.php changes to switch the datacenter in MediaWiki
    • Flip appservers-rw, api-rw, imagescaler-rw to pooled=false in the old site. After this, DNS will be changed and internal applications will start hitting the new DC

Phase 6 - apply configuration

  1. Switch the live redis configuration. This can be either scripted, or all redises can be restarted (first in the new site, then in the old one). Verify redises are indeed replicating correctly.
  2. Run puppet on the text caches in $new_site and $old_site. This starts the PII leak [TODO: check with traffic for the whole procedure]

Phase 7 - Set new site's databases to read-write

  1. Set new-site's core DB masters (shards: s1-s7, x1, es2-es3) in read-write mode.

Phase 8 - Set MediaWiki to read-write

  1. Deploy mediawiki-config wmf-config/db-$new-site.php with all shards set to read-write

Phase 9 - post read-only

  1. Start the jobqueue in the new site by running puppet there (mw_primary controls it)
  2. Run puppet on the maintenance hosts (mw_primary controls it)
  3. Update DNS records for new database masters
  4. Update tendril for new database masters
  5. Set the TTL for the DNS records to 300 seconds again.
  6. Varnish final reconfiguration:
    • Merge the second traffic puppet patch, comment-out direct route to $old_site in all mw-related cache::app_directors. This is not covered by the switchdc script
    • Run puppet on all the cache nodes in $old_site this ends the PII leak
  7. [Optional] Run the script to fix broken wikidata entities on the maintenance host of the active datacenter: sudo -u www-data mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki --force This is not covered by the switchdc script

Phase 10 - verification and troubleshooting

  1. Make sure reading & editing works! :)
  2. Make sure recent changes are flowing (see Special:RecentChanges, EventStreams, RCStream and the IRC feeds)
  3. Make sure email works (exim4 -bp on mx1001/mx2001, test an email)

Media storage/Swift

Ahead of the switchover, originals and thumbs

  1. MediaWiki: Write synchronously to both sites with https://gerrit.wikimedia.org/r/#/c/282888/https://gerrit.wikimedia.org/r/284652
  2. Cache->app: Change varnish backends for swift and swift_thumbs to point to new site with https://gerrit.wikimedia.org/r/#/c/282890/https://gerrit.wikimedia.org/r/284651
    1. Force a puppet run on cache_upload in both sites: salt -v -t 10 -b 17 -C 'G@cluster:cache_upload and ( G@site:eqiad or G@site:codfw )' cmd.run 'puppet agent --test'
  3. Inter-Cache: Switch new site from active site to 'direct' in cache::route_table for upload https://gerrit.wikimedia.org/r/#/c/282891/https://gerrit.wikimedia.org/r/284650
    1. Force a puppet run on cache_upload in new site: salt -v -t 10 -b 17 -C 'G@cluster:cache_upload and G@site:codfw' cmd.run 'puppet agent --test'
  4. Users: De-pool active site in GeoDNS https://gerrit.wikimedia.org/r/#/c/283416/https://gerrit.wikimedia.org/r/#/c/284694/ + authdns-update
  5. Inter-Cache: Switch all caching sites currently pointing from active site to new site in cache::route_table for upload https://gerrit.wikimedia.org/r/#/c/283418/https://gerrit.wikimedia.org/r/284649
    1. Force a puppet run on cache_upload in caching sites: salt -v -t 10 -b 17 -C 'G@cluster:cache_upload and G@site:esams' cmd.run 'puppet agent --test'
  6. Inter-Cache: Switch active site from 'direct' to new site in cache::route_table for upload https://gerrit.wikimedia.org/r/#/c/282892/https://gerrit.wikimedia.org/r/284648
    1. Force a puppet run on cache_upload in active site: salt -v -t 10 -b 17 -C 'G@cluster:cache_upload and G@site:eqiad' cmd.run 'puppet agent --test'

Switching back

Repeat the steps above in reverse order, with suitable revert commits

ElasticSearch

Point CirrusSearch to codfw by editing wmgCirrusSearchDefaultCluster InitialiseSettings.php. The usual default value is "local", which means that if mediawiki switches DC, everything should be automatic. For this specific switch, the value has been set to "codfw" to switch Elasticsearch ahead of time.

To ensure coherence in case of lost updates, a reindex of the pages modified during the switch can be done by following Recovering from an Elasticsearch outage / interruption in updates.

Traffic

GeoDNS user routing

Inter-Cache routing

Cache->App routing

Specifics for Switchover Test Week

After switching all applayer services we plan to switch successfully, we'll switch user and inter-cache traffic away from eqiad:

  • The Upload cluster will be following similar instructions on the 14th during the Swift switch.
  • Maps and Misc clusters are not participating (low traffic, special issues, validated by the other moves)
  • This leaves just the text cluster to operate on below:
  1. Inter-Cache: Switch codfw from 'eqiad' to 'direct' in cache::route_table for the text cluster.
  2. Users: De-pool eqiad in GeoDNS for the text cluster.
  3. Inter-Cache: Switch esams from 'eqiad' to 'codfw' in cache::route_table for the text cluster.
  4. Inter-Cache: Switch eqiad from 'direct' to 'codfw' in cache::route_table for the text cluster.

Before reversion of applayer services to eqiad, we'll revert the above steps in reverse order to undo them:

  1. Inter-Cache: Switch eqiad from 'codfw' to 'direct' in cache::route_table for all clusters.
  2. Inter-Cache: Switch esams from 'codfw' to 'eqiad' in cache::route_table for all clusters.
  3. Users: Re-pool eqiad in GeoDNS.
  4. Inter-Cache: Switch codfw from 'direct' to 'eqiad' in cache::route_table for all clusters.

Services

  • RESTBase and Parsoid already active in codfw, using eqiad MW API.
  • Shift traffic to codfw:
    • Public traffic: Update Varnish backend config.
    • Update RESTBase and Flow configs in mediawiki-config to use codfw.
  • During MW switch-over:

Other miscellaneous

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=User:Giuseppe_Lavagetto/Switch_Dc_2017&oldid=1754930"