MariaDB Upgrade

MariaDB Upgrade

Only ever upgrade a slave. A master should be demoted first.

1. Depool the box.
   • Check https://noc.wikimedia.org/dbtree/; box should disappear.
   • Watch SHOW PROCESSLIST until connections go away.
2. Disable icinga notifications for the host and it's services.
3. /etc/init.d/mysql stop
   • Watch mwlog1001:/a/mw-log/dberror.log for sudden burst of connection errors.
   • May need to manually sync-common on relevant mw[0-9]{4}.
   • a host continually failing may indicate that they are not in the right dsh group or some other issue, check it.
4. Disable puppet on the host (so it doesn't try to grab the apt db lock and break the upgrade in the next step)
5. do-release-upgrade https://wikitech.wikimedia.org/wiki/Distribution_upgrades. Afterwards:
   1. rm /etc/apt/sources.list.d/*distUpgrade
   2. rm /etc/apt/sources.list.d/wikimedia-mysql.list
   3. apt-get update
   4. dpkg -l | grep mysql | awk '{print $2}' | xargs apt-get remove -y --purge
   5. rm /etc/mysql/*cnf*
   6. rm -rf /a/sqldata/*
      • Yes, everything in the datadir. Make sure it's the right box :-)
6. Enable puppet on the host
7. In puppet, site.pp:
   • mariadb => true
   • innodb_file_per_table => true
8. puppetd --test
   • If stuff fails:

     puppet may have put the old mysql packages back on again, check if they are there and if so remove them

     now apt-get update and retry. The wikimedia-mariadb.list doesn't get added in time...

9. xtrabackup clone: https://wikitech.wikimedia.org/wiki/Setting_up_a_MySQL_replica
   • The source box should also be depooled, or at least have it's MW load reduced.
10. /etc/init.d/mysql start
    • innodb_file_per_table=1 on a shard with lots of databases/tables results in a slow startup. Wait it out.
11. Use /a/sqldata/xtrabackup_slave_info to setup replication
    • CHANGE MASTER TO MASTER_HOST=..., MASTER_PORT=3306, MASTER_USER='repl', MASTER_PASSWORD='<in private repo>', MASTER_LOG_FILE=..., MASTER_LOG_POS=...;
12. START SLAVE
    • Wait for it to catch up.
13. Repool the box
    • Use a low load setting to let it warmup (hour or so?) then raise to full steam.
14. Enable icinga notifications.