Deploying a software update fleet-wide using debdeploy and debmonitor
Each update needs to be deployed using a spec file. To generate one log into one of the cluster management hosts (
cumin1001.eqiad.wmnet, cumin2002.codfw.wmnet as your regular user and run:
It will walk you through the necessary steps. For the 'update type' pick 'tool', this documentation currently doesn't cover library rollouts.
For each distro, enter the version to update to. If an update is only available for a specific distro, simply leave version for the others empty. debdeploy will check the installed distro release during deployment and only update system which have a target version specified.
elukey@cumin1001:~$ generate-debdeploy-spec Please enter the name of source package (e.g. openssl). Leave blank or type 'quit' to abort >prometheus-memcached-exporter You can enter an optional comment, e.g. a reference to a security advisory or a CVE ID mapping > tool -> The updated packages is an enduser tool, can be rolled-out immediately. daemon-direct -> Daemons which are restarted during update, but which do no affect existing users. daemon-disrupt -> Daemons which are restarted during update, where the users notice an impact. The update procedure is almost identical, but displays additional warnings library -> After a library is updated, programs may need to be restarted to fully effect the change. In addition to libs, some applications may also fall under this rule, e.g. when updating QEMU, you might need to restart VMs. Please enter the update type: >daemon-direct Please enter the version which fixed in in jessie. Leave blank if no fix is available/required for a given distro. >0.4.1+git20181010.2fa99eb-1 Please enter the version which fixed in in trusty. Leave blank if no fix is available/required for a given distro. > Please enter the version which fixed in in stretch. Leave blank if no fix is available/required for a given distro. >0.4.1+git20181010.2fa99eb-1 Please enter a name under which the YAML file should be created >2018-10-11-prometheus-memcached-exporter.yaml
Once you have generated the spec file you can deploy it with debdeploy. Debdeploy operates on Cumin aliases, you can see the available aliases in /etc/cumin/aliases.yaml.
To e.g. deploy the update to all restbase hosts you can run:
sudo debdeploy deploy -u $SPECFILE -s restbase
You can also deploy towards complete data centers by using the aliases
codfw, etc. There's also
all. You can use your own cumin host selector as well with
debdeploy won't run
apt update, so if you uploaded the new version very recently, it won't make any changes. You can either use cumin to run
apt update on all affected hosts, or wait up to 30 minutes for it to be run automatically along with the Puppet agent.
Example of execution:
elukey@neodymium:~$ sudo debdeploy deploy -u 2018-10-11-prometheus-memcached-exporter.yaml -s memcached-codfw Rolling out prometheus-memcached-exporter: Daemon update without user impact These hosts are already up-to-date: mc[2020-2021,2026,2030,2035-2036].codfw.wmnet (6 hosts) prometheus-memcached-exporter was updated: 0.3.0+ds1-1 -> 0.4.1+git20181010.2fa99eb-1 mc[2019,2022-2025,2027-2029,2031-2034].codfw.wmnet (12 hosts)
To track which servers still need a given update you can use https://debmonitor.wikimedia.org/