Switch Datacenter/DeploymentServer

From Wikitech

This page describes the procedure to switch over the master deployment server from one host to another.


  • Disable puppet on deployment servers:

sudo cumin 'R:class = role::deployment_server' 'disable-puppet "Switchover of the deployment server"'

  • Merge a DNS change that points all the deployment CNAME to the new master hosts, see this change.
  • Change deployment_server and scap::deployment_server variables in hiera, see this change.
  • Run puppet on the new master
  • Run puppet on all the other servers
  • Using grep block_deployments /etc/scap.cfg, verify that:
    • Deployments on the old master are blocked
    • Deployments on the new master are not blocked
  • Since keyholder configuration might have changed but not been reloaded on the spare deployment server, restart the keyholder service on the new master and test it
deployXXXX:~ # sudo systemctl restart keyholder-proxy.service    deployXXXX:~ #SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -i /etc/keyholder.d/deploy_jenkins -l deploy-jenkins releases1002.eqiad.wmnet
  • Temporary workaround for task T197470: Run the following on all deployment servers after replacing deployment server url accordingly
deployXXXX:~# find /srv/deployment -name DEPLOY_HEAD | xargs sed -i "s/git_server: deploy2002.codfw.wmnet/git_server: deploy1002.eqiad.wmnet/"
  • Test scap deployment
    cd /srv/mediawiki-staging; scap sync-world "check the deployment server after switchover". This will also test helmfile deployments.
  • Test a scap3 deployment
  • email ops@ about the switch of main deployment server