Switch Datacenter/DeploymentServer
Jump to navigation
Jump to search
This page describes the procedure to switch over the master deployment server from one host to another.
Procedure
- Disable puppet on deployment servers:
sudo cumin 'R:class = role::deployment_server' 'disable-puppet "Switchover of the deployment server"'
- Merge a DNS change that points all the deployment CNAME to the new master hosts, see this change.
- Change deployment_server and scap::deployment_server variables in hiera, see this change.
- Run puppet on the new master
- Run puppet on all the other servers
- Using
grep block_deployments /etc/scap.cfg
, verify that:- Deployments on the old master are blocked
- Deployments on the new master are not blocked
- Since keyholder configuration might have changed but not been reloaded on the spare deployment server, restart the keyholder service on the new master and test it
sudo systemctl restart keyholder-proxy.service
SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -i /etc/keyholder.d/deploy_jenkins -l deploy-jenkins releases1002.eqiad.wmnet
- Test scap deployment
cd /srv/mediawiki-staging; scap sync-world "check the deployment server after switchover"
. This will also test helmfile deployments. - Test a scap3 deployment
- email ops@ about the switch of main deployment server