Obsolete:Portal:Cloud VPS/Admin/Managing package upgrades

This page contains historical information. It may be outdated or unreliable.

This workflow is no longer relevant, most package upgrades are now managed automatically with unattended-upgrades

2019

This page describes general Apt mechanics as they relate to the Wikimedia Cloud Services environment. Security updates are applied by default for all instances (without the ability to disable) but several other sources of updates exist that may require special handling. If a tenant chooses to disable unattended upgrade for package updates they will need to use the attended upgrade workflow. The intended outcome is to have a proper and common workflow to keep systems updated and sane from the distro point of view, specially with security updates.

This was being originally developed in phab:T181647.

Channels of updates

There are 3 main channels/repos for package updates. Different distro may have different channels/repos and some custom systems may have even more (for example, cloud repos). Generally speaking, there are 3 main channels, which are configured via /etc/apt/sources.list and /etc/apt/sources.list.d/*.

To know the currently configured sources, issue the command apt-cache policy.

Security updates

Usually something like this:

deb http://security.debian.org/ stretch/updates main

This repo should contain security fixes for packages in the given suite (stable). Be aware that only stable/old-stable/old-old-stable are warranted to have some kind of security support. This means that testing/unstable/experimental don't have this channel available.

Package updates coming from this repo/channel should be trusted to don't break systems, since the patch changes should be minimal and only address concrete security issues.

More info about security patches in Debian can be found at: https://security-tracker.debian.org/tracker/

Distro updates

This is usually something like this:

deb http://mirrors.wikimedia.org/debian/ stretch main

Is the main channel for any given distro. This repo should contain regular updates for packages in the given suite (stable). This is where point releases of the distro happen (including security updates), plus some transient packages (such as tzdata, and friends). Every distro should have this enabled in some form, and is the standard to obtain package upgrades for the OS.

Packages updates coming from this repo/channel should be trusted to don't break systems, patches should be minimal and stable, but generally not that small as in security.

There is additionally 'jessie-updates', 'stretch-updates' and so on. These are a replacement for the old 'volatile' approach, which contains transient packages and other small updates.

deb http://mirrors.wikimedia.org/debian/ stretch-updates main

WMF updates

Packages coming from internal WMF repositories.

deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty

See APT_repository for more details.

Workflow

This information is outdated. (last update: 2019)

Description of the intended workflow and steps for keeping systems updated. We had the idea of integrating this workflow as part of the clinic duties activities.

Gathering information

The first step is to obtain information on the status of the system and important/key packages. The question is Is there any interesting pending package upgrade?.

There is the apt-upgrade script to help in reporting pending package upgrades.

user@machine:~$ sudo apt-upgrade -u report

In Toolforge, a fleet-wide report can be obtained jumping into a clush master and runnig the script::

aborrero@machine:~$ ssh tools-clushmaster-02.tools.eqiad.wmflabs
aborrero@tools-clushmaster-02:~$ clush -w @all 'sudo apt-upgrade -u report'
[...]

There is a known issue with apt-upgrade (phab:T208863) where it fails to run due to an error in updating the apt cache. If that happens, just re-run the command.

Key packages subject to apt pinning won't be reported, so you need an external source of information like the Debian package tracker: https://tracker.debian.org/

Adjusting operations/puppet.git

There are some configuration regarding package upgrades that are persisted into operations/puppet.git.

unattended-upgrades hiera keys to enable/disable allowed origins: modules/apt/manifests/unattendedupgrades.pp
Toolforge apt pinnings: modules/toollabs/manifests/apt_pinning.pp

The latter contains apt pinning for some key packages that we don't want to be upgraded by any automatic mechanism.

Packages are usually like:

linux kernel
pam libs
kubernetes packages
nginx

Should a newer version come, a patch must increase the version pinning.

Auto upgrading packages using unattended-upgrades

The unattended-upgrades package should be able to automatically upgrade certain packages from configured origins. Usually, just by installing the unattended-ugprades package in any system will:

install a cronjob to do the thing
enable the security updates repo by default
start working

Apart of the cronjob, running the tool by hand is also possible, and this is the recommended command line for the initial touch:

user@machine:~$ sudo unattended-upgrades --dry-run -v

There are puppet hiera keys to enable/disable allowed origins for unattended-upgrades, see modules/apt/manifests/unattendedupgrades.pp.

The unattended-upgrades tool will compute the origin of each package (vs the allowed by config), check the readiness of the given package for direct upgrade (i.e, no manual intervention required) and other things. If all is OK, then the tool will update the package directly.

Be aware that this could mean arbitrary restart of some daemons and services, so care should be taken to only allow origins that are known to be harmless for package upgrades.

Explicit upgrades using apt-upgrade

Once you know which suite/channel/repo you want to upgrade, perform the actual package upgrades by means of apt-upgrade script.

user@machine01:~$ sudo apt-upgrade -un -f exclude_file.txt upgrade jessie-wikimedia
jessie-wikimedia: linux-meta 1.16 --> 1.17 [excluded]
jessie-wikimedia: linux-meta-4.9 1.16 --> 1.17 [excluded]
jessie-wikimedia: linux-image-4.9.0-0.bpo.6-amd64 [absent] --> 4.9.82-1~wmf1
jessie-wikimedia: otherpackage1 1.1.1 --> 2.2.2
jessie-wikimedia: otherpackage2 1.2.3 --> 2.3.4
commit changes? [y/N]:

In Toolforge, if enough confidence with the pending upgrades, the upgrade could be done fleet-wide with clush.

WARNING: puppet runs may open the apt/dpkg database, so 'apt-upgrade' may fail. Perhaps is good idea to disable puppet for upgrades and reenable it later

Clush operations

Some interesting clush commands:

in ubuntu machines, get all pending package upgrades, useful to inspect and search for important packages

user@tools-clushmaster-02:~$ clush -w @all -N "grep -q ubuntu /etc/os-release || exit 0 ; sudo apt-upgrade -un report" | sort -k2 | uniq

from all machines, see the list of pending packages from the 'stable' archive

user@tools-clushmaster-02:~$ clush -w @all -N "sudo apt-upgrade -un report stable" | sort -k3 | uniq

in jessie machines from the canary list, see all pending package upgrades, grouped by package

user@tools-clushmaster-02:~$ clush --hostfile=/etc/clustershell/toolforge_canary_list.txt -r -N "grep -q jessie /etc/os-release || exit 0 ; sudo apt-upgrade -u report" | sort -k3

report pending upgrades from several archives at once, grouped by package

user@tools-clushmaster-02:~$ clush -w @all -N "sudo apt-upgrade -u report | egrep trusty-tools\|jessie-tools\|stretch-tools || true" | sort -k3

FAQ

Why package X is not being upgraded by unattended-upgrades?: It is likely because it can't be cleanly upgraded (conffile conflict, unmet dependencies, the upgrade would remove other package, etc...). Run `aptitude install X` by hand to get a clue. Also, there could be an apt pinning of that package.

Does unattended-upgrades produces logs?: Yes, usually in /var/log/unattended-upgrades/

Does unattended-upgrades takes pinning/holds into account?: Yes, it should.

What happen if unattended-upgrades updates a package that requires reboot?: It can be configured to reboot the system, but obviously this is disabled by default. For example, in case of linux kernel upgrades, systems may have pending reboots to pick the updated package.

What does package pinning vs apt-mark do? (This is to provide a general understanding and not specific guidance): The 'hold mark' means: hey apt, do not touch this package *automatically*, i.e. dont install/remove/upgrade because of side depends require it. Still, if you set a hold mark for a package and then you run 'apt-get remove pkg' it will be removed. The hold mark says nothing about versioning or package origin.; Pinning establishes preferences for when calculating candidate version for upgrades. This can include pinning packages to particular versions.