Switch Datacenter/DeploymentServer

From Wikitech
Jump to: navigation, search

This page describes the procedure to switch over the master deployment server from one host to another.

Procedure

  • Merge a DNS change that points all the deployment CNAME to the new master hosts, see this change.
  • Disable puppet on deployment servers:

sudo cumin 'R:class = role::deployment_server' 'disable-puppet "Switchover of the deployment server"'

  • Change deployment_server variable in hiera, see this change.
  • Run puppet on the new master
  • Run puppet on all the other servers
  • Verify redis is being replicated, log into the new master and run echo info | redis-cli | grep role:. It should now list role:master and the former master should now list role:slave.
  • Verify the new master doesn't have the cronjob for rsyncing /srv/deployment (shown by sudo crontab -l | grep -A1 sync_deployment_dir)
  • Remove scap lock from the new master:

rm -v /var/lock/scap-global-lock