Switch Datacenter/DeploymentServer

This page describes the procedure to switch over the master deployment server from one host to another.

Procedure

• Disable puppet on deployment servers:

sudo cumin 'R:class = role::deployment_server' 'disable-puppet "Switchover of the deployment server"'

• Merge a DNS change that points all the deployment CNAME to the new master hosts, see this change.
• Change deployment_server and scap::deployment_server variables in hiera, see this change.
• Run puppet on the new master
• Run puppet on all the other servers
• Using grep block_deployments /etc/scap.cfg, verify that:
  • Deployments on the old master are blocked
  • Deployments on the new master are not blocked
• Since keyholder configuration might have changed but not been reloaded on the spare deployment server, restart the keyholder service on the new master and test it
  sudo systemctl restart keyholder-proxy.service
SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -i /etc/keyholder.d/deploy_jenkins -l deploy-jenkins releases1002.eqiad.wmnet
• Test scap deployment
  cd /srv/mediawiki-staging; scap sync-world "check the deployment server after switchover". This will also test helmfile deployments.
• Test a scap3 deployment
• email ops@ about the switch of main deployment server