Security/Debdeploy

How to assess whether we're affected

So, we've learned about a new security problem, it needs to be figured out whether the component is installed anywhere: servermon only displays packages which are flagged for updating, but doesn't maintain the full installed package status (which would be too much data for each puppet run).

This can be queried using debdeploy:

debdeploy query_version -s all --packages libfoo1 

The version number of installed packages need to be checked against the three distributions in use (Ubuntu Precise/Trusty and Debian Jessie) (the sole remaining Lucid machine has a ticket to get upgraded to jessie) In many cases the installed versions will map to the standard version in either of the three distros. Possible deviations are:

• Software not available in Debian or Ubuntu which has been packaged locally and provided by apt.wikimedia.org
• Backports of more recent package versions (which usually means that distribution security updates cannot be used as-is)
• Software with local patches applied (which usually means that the patch(es) need to be rebased on the security update)

Note that this only covers software deployed in the Debian package format. Some packages are distributed through mediawiki-services-restbase-deploy

TODO: Maybe have a central list of all software deployed which is not packaged as a DEB?

Once we have established that a vulnerable software component is in use, the impact needs to be checked. One possibility is that we're not affected at all; possible reasons are

• vulnerabilities which only affect a more recent release than the versions deployed (e.g. which a feature is affected which was introduced later)
• some vulnerabilities might already be fixed, since the security impact was discovered in hindsight
• vulnerabilities specific to Linux distributions not in use (e.g. Red Hat-specific patches) or operating systems not used on servers (e.g. vulnerabilities specific to running on Windows)

Such not-affecting issues are still useful to track, e.g. with a standard Phab ticket, so that they show up when searching for the status of a CVE ID.

If a vulnerable software is present in an affected version, the impact should be checked. The following list is not exhaustive, but lists some important vulnerability classes:

• denial of service vulnerabilities, especially in public-facing services
• vulnerabilities resulting in the execution of attacker-controlled code (e.g. by exploiting vulnerabilities in servers or indirectly in libraries used by servers)
• privilege escalation through bugs in the kernel or software with elevated privileges (while local access is limited, they could be used in combination with vulnerabilities allowing the execution of arbitrary code)
• Information disclosure of sensitive information (e.g. password hashes or cryptographic keys)

If a vulnerability is present, a Phabricator ticket should be opened (TODO: in a different project/queue than "operations" to keep queues neat and tidy?) with a brief description/assessment and a CVE ID (which allows searching for past problems more easily)

How to get the vulnerability fixed

• The version number of installed packages need to be checked against the three distributions in use. If the affected package is part of the standard version set in Ubuntu Precise/Trusty or Debian Jessie and not local modifications are applied, the status of a distribution fix needs to be assessed:
  • The Ubuntu archive is divided into four sections: main, restricted, universe and multiverse. Only main and restricted are supported by the Ubuntu security team while the other are covered by the wider Ubuntu community. For practical purposes this usually means that for universe/multiverse no security update will be provided. The status of a vulnerability can either be checked at http://people.canonical.com/~ubuntu-security/cve/$YEAR/$CVEID.html, e.g. http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-0160.html (which often contains a Launchpad bug reference)
  • In contrast Debian supports almost all packages with security updates. The exceptions are
    • Software from the non-free/contrib sections of the Debian archive
    • A small list of packages are exempted from security support since they cannot be sensibly supported with backported patches. These packages are shipped in the debian-security-support package and are mostly related to browsers or desktop-related packages. E.g. QT provides classes to display web content derived from the Webkit codebase (similar to what's used in the Safari browser), but there's no security support for QTWebkit by upstream. Usually unsupportable software is excluded from Debian releases, but in selected cases exceptions are made (as in the case of QTWebkit which is mostly used to display web content from trusted sources as opposed to a web browser processing potentially malicous HTML/Javscript. The status of upcoming security fixes can be queried in the Debian security tracker or (if not obvious) by asking for the status on #debian-security at irc.debian.org

• If the software not coming from the Debian or Ubuntu archive and has been packaged locally or is used with a patch, it's most certainly managed in operations/debs/FOO.git and the fix should be made here (with the standard git workflow in place). In most cases isolating the security fix and merging it is preferred to updating to a new upstream release (which might introduce behavioural changes). (In some cases isolating the fix might not be possible, though)

How to get the fix deployed

Now that we have a fix available, we need to get it rolled out.

Updated packages can be installed using debdeploy.

What services/programs require a restart depends very much on the updated software:

• Most daemons perform an automatic restart in postinst
• If libraries are updated, make sure to also restart applications/daemons using that software. The way the dynamic linker works on Linux is to keep older versions of a library loaded as long as existing programs use it. So, e.g. if daemon foo uses libbar 1.0-1 and the security fix 1.0-2 is installed on the system, foo will continue to use libbar 1.0-1 until it's also restarted.
• For some very low level libraries (e.g. glibc) a reboot will usually be required. The same applies to the Linux kernel.

• Contacts when updating some classes of systems
  • elastic* cluster node restarts: Erik Bernhardson/David Causse
  • analytics* hadoop restarts: Andrew Otto
• TODO: Add notes for updating specific systems etc, which systems to pick as canaries, run which initial tests

• TODO: Add Icinga checks to detect services/programs which need to be restarted (src:checkrestart can be used as the base tool)