Network leaf maintenance

Unlike spine switches, leaf switch are not redundant, so during maintenance, all servers connected to that switch are going to loose network connectivity for a duration depending on the type of maintenance.

This doc aims at streamlining this process.

Preparation

Run the sudo cookbook sre.network.depool-rack --site <SITE NAME> --rack <RACK NAME> --show --teams depool cookbook. For example sre.network.depool-rack --site codfw --rack A3 --show --teams depool

--show won't proceed with the depool, only show what needs to happen, --teams will group the servers per teams for easier notification.

Open a tracking task (for example T426199) and add the output of the cookbook to the task description, you can format it a bit better for easier readability. Tag the local DCops as well as the relevant teams.

Maintenance

Notify SREs on IRC that the maintenance is starting.

Downtime the servers located in the rack, as well as the switch itself, for example:

sudo cookbook sre.hosts.downtime -r "Switch maintenance" -t XXXX -H 2 'P{P:netbox::host%location ~ "<location>"}'

sudo cookbook sre.hosts.downtime -r "Switch maintenance" -t XXX -H 2 --force '<hostname>,<hostname> IPv6,<hostname>.mgmt'

<location> here can for example be A3.*codfw for all the servers in codfw rack A3. You can test the selector using cumin : sudo cumin 'P{P:netbox::host%location ~ "<location>"}'

Note that the depool-rack cookbook can take care of the downtime (see below).

For POPs, additional services might need to be downtimed manually (eg. LVS VIPs) : in the https://alerts.wikimedia.org/ dashboard, create a new Silence with "site=eqsin" and "severity=page" (for example)

Re-run the sre.network.depool-rack cookbook this time without --show --teams but with --downtime. If any issue with a depool, try manually or reach out to the service owners if unsure (but ideally this is figured out ahead of time).

Depending on the device vendor or type of maintenance, follow the proper runbook:

• Juniper switch upgrade
• Nokia Networking/Nokia switch upgrade (this needs to be created)
• Eventually a runbook too about switch replacement for RMA

Cleanup

At this point the switch is back up on a healthy state.

Delete the previously set downtime:

sudo cookbook sre.hosts.remove-downtime 'P{P:netbox::host%location ~ "<location>"}'

sudo cookbook sre.hosts.remove-downtime --force '<hostname>,<hostname> IPv6,<hostname>.mgmt'

If no alerts, manually repool the services where needed (this will eventually be done by the cookbook). Notify service owners that they can repool their services if they drained them.

announce the end of the maintenance.