Switch Datacenter/planned db maintenance

2020 Switch Datacenter

Important dates

• Switchover date: Tuesda, September 1st 2020: 14:00 UTC
• Switchback date: TBD

Failover schedule

• 27th Aug - LAST DB maintenance day in all DCs
• 27th Aug - Enable replication codfw -> eqiad
• 27th&28th Aug - Review codfw weights. In particular make sure section distribution and loads are ok
• 27th,28th,29th,31st Aug - Definitive Warmup (pc, es, maybe some big tables on big wikis)
• 1st Sept - 14:00 UTC Switchover to codfw support and monitoring
• 3th Sept - Disconnect eqiad -> eqiad replication
• 3th Sept - Maintenance can start on eqiad
• TBD - Enable eqiad -> codfw replication
• TBD - Review eqiad weights
• TBD - Enable eqiad -> codfw replication
• TBD - 14:00 UTC Switchover support and monitoring

Tasks

• META task for all the tasks: https://phabricator.wikimedia.org/T243316
  • Schema change on production for increase the size of wbt_text_in_lang.wbxl_language https://phabricator.wikimedia.org/T237120
  • Switchover s8 primary database master db1109 -> db1104 https://phabricator.wikimedia.org/T239238
  • Compress new Wikibase tables https://phabricator.wikimedia.org/T232446
  • Normalise MW Core database language fields length https://phabricator.wikimedia.org/T253276
  • Compress enwiki InnoDB tables https://phabricator.wikimedia.org/T254462
  • Apply updates for MCR https://phabricator.wikimedia.org/T238966
  • pl_namespace index on pagelinks is unique only in s8 https://phabricator.wikimedia.org/T256685
  • Schema change to make change_tag.ct_rc_id https://phabricator.wikimedia.org/T259831
  • Host that needs rebooting for kernel upgrade https://phabricator.wikimedia.org/T261389
  • Failover DB masters in row D https://phabricator.wikimedia.org/T186188

2018 Switch Datacenter

Important dates

• Switchover date: Wednesday, September 12th 2018: 14:00 UTC
• Switchback date: Wednesday, October 10th 2018: 14:00 UTC

Failover schedule

• 5th Sept - LAST DB maintenance day in all DCs
• 6th Sept - Enable replication codfw -> eqiad
• 6th Sept - Review db-codfw.php. In particular make sure section distribution and loads are ok
• 11th Sept - Definitive Warmup (pc, es, maybe some big tables on big wikis)
• 12th Sept - 14:00 UTC Switchover to codfw support and monitoring
• 4th Oct - LAST DB maintenance day on eqiad
• 5th Oct - Review db-eqiad.php
• 8th Oct - Enable eqiad -> codfw replication
• 10th Oct - 14:00 UTC Switchover support and monitoring

Tasks

• META task for all the tasks: https://phabricator.wikimedia.org/T189107
  • UPGRADE MARIADB on eqiad masters: https://phabricator.wikimedia.org/T204311
  • SCHEMA CHANGE (only 6 masters pending): Fix WMF schemas to not break when comment store goes WRITE_NEW https://phabricator.wikimedia.org/T187089
  • UPGRADE MARIADB: Upgrade db1062 and db1075 MARIADB: https://phabricator.wikimedia.org/T181777
  • UPGRADE SOCKET LOCATION: Upgrade db1062 and db1075 socket location https://phabricator.wikimedia.org/T148507
    • Pending only db1073 (m5 master) - stalled waiting for the network maintenance
  • UPGRADE KERNELS: Upgrade kernels in eqiad masters (or "difficult hosts"): https://phabricator.wikimedia.org/P7510 https://phabricator.wikimedia.org/P6592 https://phabricator.wikimedia.org/T184267
  • SCHEMA CHANGE (only masters pending): Make several mediawiki table fields unsigned ints on wmf databases https://phabricator.wikimedia.org/T89737
  • SCHEMA CHANGE (Only s4 master pending): Drop eu_touched in production https://phabricator.wikimedia.org/T144010
  • SCHEMA CHANGE: Deploy schema change for adding numeric primary key to wbqc_constraints table https://phabricator.wikimedia.org/T189101
  • SCHEMA CHANGE: Remove partitioning from metawiki.pagelinks from s7 masters (eqiad and codfw): https://phabricator.wikimedia.org/T203548
  • RECLONE HOST: Reclone db1114 (s1 api): https://phabricator.wikimedia.org/T203565
  • RECLONE HOST: Reclone db2054 and db2068 (s7): https://phabricator.wikimedia.org/T204127
  • (DECIDED NOT TO BE DONE) Failover masters on row D (stalled task as per the network issues): https://phabricator.wikimedia.org/T186188

2017 Switch Datacenter

Important dates

• Switchover date: Wednesday April 19th, 14:00 UTC
• Switchback date: Wednesday May 3rd, 14:00 UTC

Failover schedule

• 7th Apr - Warm up test for es
• 12th Apr - LAST DB maintenance day in all DCs
• 12th Apr - Review db-codfw.php
• 18th Apr - Definitive Warmup
• 19th -14:00 UTC Switchover to codfw support and monitoring
• 28th Apr - Review db-eqiad.php
• 28th Apr - LAST DB maintenance day on eqiad
• 3rd May -14:00 UTC Switchover support and monitoring

Tasks

• ALTER eqiad shards: https://phabricator.wikimedia.org/T130067

ALTER TABLE watchlist ADD COLUMN wl_id int unsigned NOT NULL AUTO_INCREMENT FIRST, ADD PRIMARY KEY (wl_id);

• ALTER eqiad shards: https://phabricator.wikimedia.org/T147166

ALTER TABLE tag_summary ADD COLUMN ts_id INT UNSIGNED NOT NULL AUTO_INCREMENT FIRST, ADD PRIMARY KEY (ts_id);
ALTER TABLE change_tag ADD COLUMN ct_id INT UNSIGNED NOT NULL AUTO_INCREMENT FIRST, ADD PRIMARY KEY (ct_id);

• ALTER eqiad enwiki master: https://phabricator.wikimedia.org/T132416

./osc_host.sh --host=db1052.eqiad.wmnet --port=3306 --db=enwiki --table=revision --method=ddl --no-replicate "drop key rev_id, drop primary key, add primary key (rev_id), add key rev_page_id (rev_page,rev_id)"

• ALTER eqiad s4 master: https://phabricator.wikimedia.org/T73563

ALTER TABLE filearchive MODIFY COLUMN fa_minor_mime varbinary(100) default "unknown";
ALTER TABLE image MODIFY COLUMN img_minor_mime varbinary(100) NOT NULL default "unknown";
ALTER TABLE oldimage MODIFY COLUMN oi_minor_mime varbinary(100) NOT NULL default "unknown";

• ALTER s5 wikidatawiki wb_terms table on eqiad hosts (and codfw ones codfw is back in standby): https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548

• Remove partitions from s7 metawiki.pagelinks eqiad master: https://phabricator.wikimedia.org/T153300

osc_host.sh --host=db1041 --port=3306 --db=metawiki --table=pagelinks --method=ddl --no-replicate "remove partitioning"
(several)

• Extra PRIMARY key additions (TBD) https://phabricator.wikimedia.org/T17441
• Master failovers: https://phabricator.wikimedia.org/T162133
  • Spread new masters in different racks: https://phabricator.wikimedia.org/T163895
• There's a few jessie-based mysql master servers which are still on Linux 3.19: es1011, es1014, It would be great to reboot these to Linux 4.4 (or ideally 4.9 right away)
• Upgraded pending 10.0.22 es hosts (es1011, es1014 and es1015) to mariadb 10.0.28