MariaDB/Decommissioning a DB Host

From Wikitech
Jump to navigation Jump to search

Prerequisites:

  • SSH access to one of the cluster management hosts (cumin1001.eqiad.wmnet, cumin2002.codfw.wmnet) to depool + run the decommissioning script
  • SSH access to puppetmaster1001.eqiad.wmnet to merge puppet changes
  • Access to Pwstore
  • Git repositories cloned to your host:

Decommissioning workflow:

Create a tracking ticket

  1. Create a decommission ticket with the following template: https://phabricator.wikimedia.org/maniphest/task/edit/form/52/
  2. If there is hardware problems, please specify so for the DCOps to label it so we do not re-use broken pieces.

Depool the host

  1. SSH to one of the cluster management hosts (cumin1001.eqiad.wmnet, cumin2002.codfw.wmnet)
  2. dbctl instance HOSTNAME depool && dbctl config commit -m "Depool db1091 TASKNUMBER"

Remove the host from dbctl

  1. Create a puppet patch (example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/638343)
  2. SSH to puppetmaster1001
  3. sudo puppet-merge - if you see any changes other than yours here, contact the owners to see if these are ok to merge
  4. SSH to one of the cluster management hosts (cumin1001.eqiad.wmnet, cumin2002.codfw.wmnet)
  5. sudo dbctl config commit -m "Remove HOSTNAME from dbctl TASKNUMBER"

Remove all other puppet entries

  1. Create a puppet patch (example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/638352)
    1. Changes to dhcp are no longer needed, so no need to edit: linux-host-entries.ttyS1-115200
  2. DO NOT merge the patch yet

Run the decommissioning script

  1. SSH to one of the cluster management hosts (cumin1001.eqiad.wmnet, cumin2002.codfw.wmnet)
  2. Start a screen or tmux session
  3. sudo cookbook sre.hosts.decommission -t TASKNUMBER HOSTNAME.DC.wmnet
  4. Enter console password from Pwstore

Merge puppet change

  1. SSH to puppetmaster1001
  2. sudo puppet-merge - if you see any changes other than yours here, contact the owners to see if these are ok to merge

Remove host from zarcillo

  1. Log the action in IRC (#wikimedia-operations) - !log Removing HOSTNAME from zarcillo TASKNUMBER
  2. SSH to one of the cluster management hosts (cumin1001.eqiad.wmnet, cumin2002.codfw.wmnet)
  3. sudo -i
  4. Zarcillo
    1. db-mysql db1215 -A zarcillo
    2. Execute the following queries in the MySQL prompt (remember about the semicolon):
      1. set binlog_format='ROW';
      2. delete from servers where hostname like 'HOSTNAME%';
      3. delete from instances where name like 'HOSTNAME%'; (INSTANCE is normally HOSTNAME or HOSTNAME:PORT)
      4. delete from section_instances where instance like 'HOSTNAME%';

Remove host from orchestrator

Orchestrator will purge the host automatically within 1-2 weeks but to avoid that delay it should be removed manually
  1. From the GUI (admin users only)
  2. From the CLI:
    1. Log the action in IRC (#wikimedia-operations) - !log Removing HOSTNAME from orchestrator TASKNUMBER
    2. SSH to dborch1001.wikimedia.org
    3. Single-instance host: sudo orchestrator -c forget -i HOSTNAME:3306 (use the FQDN for the HOSTNAME)
    4. Multi-instance host: sudo orchestrator -c forget -i HOSTNAME:PORT for each HOSTNAME:PORT combination (use the FQDN for the HOSTNAME)

Update the task and send it to dcops

  1. mark all the steps for "step for service owners" on: https://phabricator.wikimedia.org/T267088
  2. Reassign:
    • for eqiad to wiki_willy
    • for codfw to wiki_willy
  3. Remove #DBA tag and add #dc-ops and #ops-eqiad OR #ops-codfw.
  4. Add the following comment: "This host is ready for DC-Ops to decommission".



This page is a part of the SRE Data Persistence technical documentation
(go here for a list of all our pages)

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=MariaDB/Decommissioning_a_DB_Host&oldid=2131576"