Deploying a software update fleet-wide using debdeploy and debmonitor

Each update needs to be deployed using a spec file. To generate one log into one of the cluster management hosts (cumin1002.eqiad.wmnet, cumin2002.codfw.wmnet as your regular user and run:

 generate-debdeploy-spec

It will walk you through the necessary steps. For the 'update type' pick 'tool', this documentation currently doesn't cover library rollouts.

For each distro, enter the version to update to. If an update is only available for a specific distro, simply leave version for the others empty. debdeploy will check the installed distro release during deployment and only update system which have a target version specified.

elukey@cumin1001:~$ generate-debdeploy-spec
Please enter the name of source package (e.g. openssl). Leave blank or type 'quit' to abort
>prometheus-memcached-exporter
You can enter an optional comment, e.g. a reference to a security advisory or a CVE ID mapping
>
tool           -> The updated packages is an enduser tool, can be
                  rolled-out immediately.
daemon-direct  -> Daemons which are restarted during update, but which
                  do no affect existing users.
daemon-disrupt -> Daemons which are restarted during update, where the
                  users notice an impact. The update procedure is almost
                  identical, but displays additional warnings
library        -> After a library is updated, programs may need to be
                  restarted to fully effect the change. In addition
                  to libs, some applications may also fall under this rule,
                  e.g. when updating QEMU, you might need to restart VMs.
Please enter the update type:
>daemon-direct
Please enter the version which fixed in in jessie. Leave blank if no fix is available/required for a given distro.
>0.4.1+git20181010.2fa99eb-1
Please enter the version which fixed in in trusty. Leave blank if no fix is available/required for a given distro.
>
Please enter the version which fixed in in stretch. Leave blank if no fix is available/required for a given distro.
>0.4.1+git20181010.2fa99eb-1
Please enter a name under which the YAML file should be created
>2018-10-11-prometheus-memcached-exporter.yaml

Once you have generated the spec file you can deploy it with debdeploy. Debdeploy operates on Cumin aliases, you can see the available aliases in /etc/cumin/aliases.yaml.

To e.g. deploy the update to all restbase hosts you can run:

 sudo debdeploy deploy -u $SPECFILE -s restbase

You can also deploy towards complete data centers by using the aliases eqiad, codfw, etc. There's also all. You can use your own cumin host selector as well with -Q.

Note that debdeploy won't run apt update, so if you uploaded the new version very recently, it won't make any changes. You can either use cumin to run apt update on all affected hosts, or wait up to 30 minutes for it to be run automatically along with the Puppet agent.

Example of execution:

elukey@neodymium:~$ sudo debdeploy deploy -u 2018-10-11-prometheus-memcached-exporter.yaml -s memcached-codfw
Rolling out prometheus-memcached-exporter:
Daemon update without user impact

These hosts are already up-to-date:
  mc[2020-2021,2026,2030,2035-2036].codfw.wmnet (6 hosts)

prometheus-memcached-exporter was updated: 0.3.0+ds1-1 -> 0.4.1+git20181010.2fa99eb-1
  mc[2019,2022-2025,2027-2029,2031-2034].codfw.wmnet (12 hosts)

To track which servers still need a given update you can use https://debmonitor.wikimedia.org/