APT repository

From Wikitech
Jump to navigation Jump to search

Wikimedia maintains its own APT repository available at http://apt.wikimedia.org/wikimedia and currently hosted on install1002.

This repository contains Debian and Ubuntu packages modified for use by Wikimedia as well as packages for Wikimedia-originated projects.

Repository Structure

Starting with Debian 9 Stretch, we are using a new repository structure with the following components:

  • main contains all the packages that we either create internally or packages that we backport or modify from Debian and which are useful fleet-wide (e.g., Icinga plugins, Cumin, etc.)
  • thirdparty/hwraid contains binary-only drivers for RAID management used on baremetal servers only.
  • All other components using the thirdparty/ prefix are synchronised from external repositories (e.g., thirdparty/cloudera, thirdparty/confluent, thirdparty/ci (Jenkins) and thirdparty/k8s (Docker)). Please do not add internally built packages to thirdparty.
  • Some packages should not be available fleet-wide, but rather be used in more specific scenarios. One example is component/ci, which provides co-installable PHP packages that are only necessary for CI.

You can find a complete list of repository components in our Puppet config

Older Distributions

For older Linux distributions like Debian 8 Jessie or Ubuntu 14.04 Trusty, the new repository organization was not applied retroactively due to the extra effort required.

In such cases, internally built packages could be in main or backports. If you add new packages, always use main.

Additionally, third-party packages are not fully split into logical components and they could be in the thirdparty top-level component as well as in sub-component thirdparty/foo.

If you are updating third-party packages, remember to update either thirdparty or the thirdparty/foo sub-component depending on where they were originally located. If you can, a better long-term plan is to update to Debian 9 Stretch.

Finally, Ubuntu 14.04 Trusty also has universe and non-free. Ignore those and use main.

External Access

For Wikimedia servers and Cloud VPS instances, the repositories are automatically configured via Puppet.

To use this repository from an external host, the following lines need to be present in /etc/apt/sources.list or /etc/apt/sources.d/wikimedia.list:

## Wikimedia APT repository
deb http://apt.wikimedia.org/wikimedia stretch-wikimedia main
deb-src http://apt.wikimedia.org/wikimedia stretch-wikimedia main

Use the right distribution, depending on which Debian/Ubuntu version was installed (e.g. stretch-wikimedia, jessie-wikimedia or trusty-wikimedia).

Here is a complete list of all the available sources:

deb http://apt.wikimedia.org/wikimedia trusty-wikimedia main experimental backports
deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main experimental backports
deb http://apt.wikimedia.org/wikimedia stretch-wikimedia main
deb-src http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports
deb-src http://apt.wikimedia.org/wikimedia trusty-wikimedia main
deb-src http://apt.wikimedia.org/wikimedia stretch-wikimedia main

Additionally, to make sure that the system prefers packages from this repository, and not packages from the origin distributions even if they have a higher version number. For example, this APT source is "pinned" with a higher priority in /etc/apt/preferences.d/wikimedia.pref:

Package: *
Pin: release o=Wikimedia
Pin-Priority: 1001

You can confirm Wikimedia's APT repository is taking preference now by running `apt-cache policy <package>`:

$ apt-cache policy puppetmaster
puppetmaster:
  Installed: (none)
  Candidate: 4.8.2-5
  Version table:
     4.8.2-5 1001
        500 http://deb.debian.org/debian stretch/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia stretch-wikimedia/main amd64 Packages

Security

The Wikimedia repository is signed with the Wikimedia Archive Automatic Signing Key <root@wikimedia.org>. This public key must be installed in APT's GPG keyring so it can verify packages successfully.

The key can be retrieved from /Stretch-Key or /Key (TODO: Clarify why there are two keys).

To install it, download the key and add it using `apt-key`:

# Debian Stretch
$ wget -O wikimedia-apt-key "https://wikitech.wikimedia.org/w/index.php?title=APT_repository/Stretch-Key&action=raw"
$ apt-key add wikimedia-apt-key
OK

# Debian Jessie / Ubuntu Trusty
$ wget -O wikimedia-apt-key "https://wikitech.wikimedia.org/w/index.php?title=APT_repository/Key&action=raw"
$ apt-key add wikimedia-apt-key
OK

You can also retrieve the key from a Cloud VPS host: ssh <host> apt-key export root@wikimedia.org

See also

  • pbuilder for building packages for different distribution versions and architectures
  • reprepro for additional information on how to use this tool
  • Wikimedia APT browser for a simple web package listing