SRE/Service Operations/Documentation/Reboots
There are times when we need to reboot our fleet. Here are some notes to help us go through it like
Datastores
Memcache cluster
Servers of this cluster need no depooling of any sort, and we have a pool of servers to pick up the traffic of any unavailable server. The cached data in the memecache cluster are crucial for our latency, so it is highly recommended to reboot 1 server at a time per DC, with a sleep time of 15'-20' minimum between reboots, allowing the server to warm up.
Cookbook: sre.memcached.roll-reboot-restart
Use the cookbook sre.memcached.roll-reboot-restart to perform rolling reboots or daemon restarts on memcached hosts.
The cookbook automatically:
- Verifies the gutter pool is healthy before touching the main pool (as failover capacity)
- Confirms memcached is active on each host after the operation
Options
Standard cookbook options are used, with the addition of --min-uptime, so to provide some easy way to resume operations.
- Currently the
--queryis broken, please only use aliases --min-uptime: Only include hosts whose memcached service has been running at least this long (e.g.7d,24h,604800)
Examples
Rolling reboot of eqiad, one host at a time with 15-minute sleep (recommended defaults):
cookbook sre.memcached.roll-reboot-restart --reason "Debian reboots" \
--alias memcached-eqiad --batchsize 1 --grace-sleep 900 reboot
Restart only hosts whose memcached has been running for at least 10 days (e.g. to resume interrupted rolling restarts):
cookbook sre.memcached.roll-reboot-restart --reason "Resume reloads" \
--alias memcached-eqiad --min-uptime 10d --batchsize 2 restart_daemons
Redis misc
Redis hosts can be rebooted one after the other, taking care of waiting for replication to be back up after the reboot. Dashboards: [redis-misc on Grafana] Please do not restart these servers during deployments because docker-registry uses redis
Service Upgrade
Upgrading redis-server
A simple apt-get install -y redis-server is enough, but it will not trigger a restart, as we are using one systemd service per port, thus you will see the following message
redis-server.service is a disabled or a static unit not running, not starting it.`
Restarting instances
Always restart replicas first, then primaries second. Restart all instances one by one with a 5-second delay:
rdb1001:# for port in 6378 6379 6380 6381 6382; do echo "Restarting $port..."; systemctl restart redis-instance-tcp_$port; sleep 5; done
Server reboot
Run the sre.hosts.reboot cookbook, taking care to reboot the replica first, and to wait for the replication status to be ok before moving to the primary.
Common checks
Checking replication status
To check the replication status of all instances:
rdb1001:# for port in 6378 6379 6380 6381 6382; do echo "--- $port ---"; redis-cli -p $port -a $(grep requirepass /etc/redis/tcp_$port.conf | awk '{print $2}') --no-auth-warning info replication | grep -E "master_link_status|master_last_io_seconds_ago|master_sync_in_progress"; echo "-----"; done
Example output:
--- 6379 ---
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:1
-----
--- 6380 ---
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
-----
A healthy, caught-up replica should show:
master_link_status:upmaster_last_io_seconds_ago: expect a low value (0-10)master_sync_in_progress:0
Full resync after restart
After a restart, the replica will perform a full resync since it has no cached master state:
Master replied to PING, replication can continue...
Partial resynchronization not possible (no cached master)
Full resync from master: 7fe8b694f6afc193b57187a779dfd037c9884ee3:0
During this process, master_last_io_seconds_ago:-1 indicates the replica has not yet connected or is still performing a full sync.
Note: In the case of a primary redis server restart with a large dataset, it may take a while for replicas to reconnect, as the primary will need to load the full dataset into memory first.
docker-registry
docker-registry doesn't use HA for redis, so the reboot of its redis nodes will cause a short unavailability. Avoid restarting these nodes during deployments, as the unavailability of the docker-registry redis could cause deployment failures.
Etcd
Kubernetes
Production
Workers
sudo cookbook -d sre.k8s.reboot-nodes --batchsize 15 --k8s-cluster wikikube-codfw --reason "Reason" --alias wikikube-worker-codfw --minimal-cordon
sudo cookbook -d sre.k8s.reboot-nodes --batchsize 15 --k8s-cluster wikikube-eqiad --reason "Reason" --alias wikikube-worker-eqiad --minimal-cordon
Control plane
sudo cookbook -d sre.k8s.reboot-nodes --batchsize 1 --k8s-cluster wikikube-eqiad --reason "Reason" --alias wikikube-master-eqiad --minimal-cordon
Dragonfly supernodes
Lock scap to avoid overloading the registry servers with a big deployment
deploy1003:~$ scap lock --all "Dragonfly supernodes reboot"
Then reboot using the sre.hosts.reboot-single cookbook
Staging
Poolcounter
Each poolcounter server should be removed from mediawiki-config/wmf-config/ProductionServices.php with a gerrit patch before reboot, then that patch should be deployed to production using scap backport.
After the reboot, add the server back and remove the next server in the same patch, scap backport it, and repeat until all poolcounter servers are rebooted and back in mediawiki-config/wmf-config/ProductionServices.php
Thumbor uses poolcounter as well, but will fail open if its poolcounter server is down. You can forgo swapping the servers out in thumbor's deployment-charts/helmfile.d/services/thumbor/values-{eqiad,codfw}.yaml files if the interruption is short.
Chartmuseum
Chartmuseum is addressed through helm-charts.discovery.wmnet. It is backed by one VM in each datacentre.
# Depool codfw
sudo confctl --object-type discovery select 'dnsdisc=helm-charts.*,name=codfw' set/pooled=false
# Reboot codfw
sudo cookbook sre.hosts.reboot-single -r "May 2025 Reboots" chartmuseum2001.codfw.wmnet
# Repool codfw
sudo confctl --object-type discovery select 'dnsdisc=helm-charts.*,name=codfw' set/pooled=true
# Rinse and repeat for eqiad
sudo confctl --object-type discovery select 'dnsdisc=helm-charts.*,name=eqiad' set/pooled=false
sudo cookbook sre.hosts.reboot-single -r "May 2025 Reboots" chartmuseum1001.eqiad.wmnet
sudo confctl --object-type discovery select 'dnsdisc=helm-charts.*,name=eqiad' set/pooled=true