Primary database switchover

See Switch master/Archive for the previous version.

Updates

These parts need updating in case of a switchover:

dbctl (for mediawiki config)
puppet
DNS
heartbeat
orchestrator tags
zarcillio
query killer

Steps

There are previous steps before doing the topology change, use this task as reference: https://phabricator.wikimedia.org/T313383

The key part is to first change the topology to get all the hosts to replicate from the new master: Assuming old master: db1181 and new master: db1136

sudo db-switchover --timeout=25 --only-slave-move db1181 db1136

Once that is finished, we are ready to get the section into read-only and then change the master:

To interact with mediawiki dependencies we use dbctl:

sudo dbctl --scope eqiad section sX ro "Maintenance until 06:15 UTC - TXXXX"
sudo dbctl config commit -m "Set sX eqiad as read-only for maintenance - TXXXX"

Now we can perform the switch:

sudo db-switchover --skip-slave-move db1181 db1136

Assuming the above went fine, we can set the section back to WR and promote the new master

sudo dbctl --scope eqiad section sX set-master db1136
sudo dbctl --scope eqiad section sX rw
sudo dbctl config commit -m "Promote db1136 to sX primary and set section read-write TXXXX"

The rest of the steps provided db-switchover by the after the switchover need to be followed in order to ensure that the post-switchover tasks are completed successfully. Check this task for reference: https://phabricator.wikimedia.org/T313383

Please also see the main dbctl article