Fundraising/techops/procedures/servers-standard software vulnerability management
Operating System and Vendor Software Vulnerability Procedures
Purpose
This procedure document applies to off-the-shelf software provided by Debian, mariadb.org, GrafanaLabs and other vendors installed on servers in the Fundraising environment. Bespoke and locally modified software is handled by FR Tech and following this procedure.
Vulnerability Tracking
FR Tech Ops SREs should monitor security mailing lists for vulnerability announcements for software we use. Relevant lists include:
- Debian: debian-security-announce@lists.debian.org
- MariaDB: announce@lists.mariadb.org
- Mediawiki, pybal, general: security@wikimedia.org
- CiviCRM: security@civicrm.org
- General: Bugtraq, US-CERT, fulldisclosure, oss-security
Automated Tracking
The pxeboot-preseed-puppet server build process configures apt repositories and modifies the package selection appropriate for the hardware type and server role. In cases where non-stock packages are used, puppet configures the relevant apt repository and uses apt-pin to ensure that only the required packages are installable from the non-Debian repository. Thus for all installed deb packages, standard package management tools show vendor updates as soon as they become available for install.
Most Fundraising servers run unattended-upgrades daily and are configured to apply patches automatically, except for cases where a package update could cause an outage.
Fundraising servers also run package_update_check daily, which uses a simulated apt dist-upgrade to check for available updates. This script notifies SREs if intervention is required. Fundraising Tech Ops SREs monitor this mailing list daily for maintenance tasks.
Vulnerability Management
For security updates, FR Tech Ops should evaluate the vulnerability based on the risk ranking provided the software vendor according to WMFs (draft) Security Policy - Vulnerability Matrix.
When a patch can be applied without significant service disruption, it is typically done automatically within a day by unattended-upgrades. Otherwise FR Tech Ops is notified daily by each server with outdated packages requiring intervention.
Any update that can be done without significant interruption should be performed as soon as possible, preferably the same day the notification is first received. In this case, no additional tracking is required.
If a patch requires significant service disruption, or in cases where other mitigation is required, a private Phabricator task should be opened to document the vulnerability, risk, and technical assessment, and to plan and track followup actions.
In all cases, FR Tech Ops is required to apply patches and mitigations within the time period specified in WMFs (draft) Security Policy - Vulnerability Matrix for the assessed risk level. This policy has shorter timelines than PCI DSS.
The preferred method to apply patches and updates is to run risky_package_updater on each server. This script automates apt and puppet steps, and also checks for required service restarts.