Primary database switchover

From Wikitech
(Redirected from Master switch)
Jump to navigation Jump to search

See Switch master/Archive for the previous version.

Checklist

See an example task and checklist at: https://phabricator.wikimedia.org/T313383

From the operations/software/ repo, run the following script to get a proper checklist of all the steps needed:

python3 dbtools/switchover-tmpl.py --dc eqiad --old $OLDMASTERHOSTNAME --ticket TXXXXX $SECTION $NEWMASTERHOSTNAME
  • $SECTION and $NEWMASTERHOSTNAME are required, dc and old master would be automatically looked up if omitted.

Updates

These parts need updating in case of a switchover:

  • dbctl (for mediawiki config)
  • puppet
  • DNS
  • heartbeat
  • orchestrator tags
  • zarcillio
  • query killer

Steps

The key part is to first change the topology to get all the hosts to replicate from the new master: Assuming old master: db1181 and new master: db1136

sudo db-switchover --timeout=25 --only-slave-move db1181 db1136

Once that is finished, we are ready to get the section into read-only and then change the master:

To interact with mediawiki dependencies we use dbctl:

sudo dbctl --scope eqiad section sX ro "Maintenance until 06:15 UTC - TXXXX"
sudo dbctl config commit -m "Set sX eqiad as read-only for maintenance - TXXXX"

Now we can perform the switch:

sudo db-switchover --skip-slave-move db1181 db1136

Assuming the above went fine, we can set the section back to WR and promote the new master

sudo dbctl --scope eqiad section sX set-master db1136
sudo dbctl --scope eqiad section sX rw
sudo dbctl config commit -m "Promote db1136 to sX primary and set section read-write TXXXX"

The rest of the steps provided switchover-tmpl.py by the after the switchover need to be followed in order to ensure that the post-switchover tasks are completed successfully.

Please also see the main dbctl article