APT repository

From Wikitech
Jump to navigation Jump to search

Wikimedia maintains its own APT repository available at https://apt.wikimedia.org/wikimedia/ and currently hosted on apt1001.

This repository contains Debian and Ubuntu packages modified for use by Wikimedia as well as packages for Wikimedia-originated projects.

Repository Structure

Starting with Debian 9 Stretch, we are using a new repository structure with the following components:

  • main contains all the packages that we either create internally or packages that we backport or modify from Debian and which are useful fleet-wide (e.g., Icinga plugins, Cumin, etc.)
  • thirdparty/hwraid contains binary-only drivers for RAID management used on baremetal servers only.
  • All other components using the thirdparty/ prefix are synchronised from external repositories (e.g., thirdparty/cloudera, thirdparty/confluent, thirdparty/ci (Jenkins) and thirdparty/k8s (Docker)). Please do not add internally built packages to thirdparty.
  • Some packages should not be available fleet-wide, but rather be used in more specific scenarios such as the following:
    • component/ci provides co-installable PHP packages that are only necessary for CI
    • "contain" specific packages that depend on outdated or backported libraries (e.g. crypto libraries)
    • ease migrations and upgrades for certain roles, while leaving the rest of the fleet untouched

You can find a complete list of repository components in our Puppet config

Older Distributions

For older Linux distributions like Debian 8 Jessie or Ubuntu 14.04 Trusty, the new repository organization was not applied retroactively due to the extra effort required.

In such cases, internally built packages could be in main or backports. If you add new packages, always use main.

Additionally, third-party packages are not fully split into logical components and they could be in the thirdparty top-level component as well as in sub-component thirdparty/foo.

If you are updating third-party packages, remember to update either thirdparty or the thirdparty/foo sub-component depending on where they were originally located. If you can, a better long-term plan is to update to Debian 9 Stretch.

Finally, Ubuntu 14.04 Trusty also has universe and non-free. Ignore those and use main.

External Access

For Wikimedia servers and Cloud VPS instances, the repositories are automatically configured via Puppet.

To use this repository from an external host, the following lines need to be present in /etc/apt/sources.list or /etc/apt/sources.list.d/wikimedia.list: 

## Wikimedia APT repository
deb http://apt.wikimedia.org/wikimedia stretch-wikimedia main
deb-src http://apt.wikimedia.org/wikimedia stretch-wikimedia main

Use the right distribution, depending on which Debian/Ubuntu version was installed (e.g. bullseye-wikimedia, buster-wikimedia or trusty-wikimedia).

Here is a complete list of all the available sources: 

deb http://apt.wikimedia.org/wikimedia trusty-wikimedia main experimental backports
deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main experimental backports
deb http://apt.wikimedia.org/wikimedia stretch-wikimedia main
deb http://apt.wikimedia.org/wikimedia buster-wikimedia main
deb http://apt.wikimedia.org/wikimedia bullseye-wikimedia main
deb-src http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports
deb-src http://apt.wikimedia.org/wikimedia trusty-wikimedia main
deb-src http://apt.wikimedia.org/wikimedia stretch-wikimedia main
deb-src http://apt.wikimedia.org/wikimedia buster-wikimedia main
deb-src http://apt.wikimedia.org/wikimedia bullseye-wikimedia main

Additionally, to make sure that the system prefers packages from this repository, and not packages from the origin distributions even if they have a higher version number. For example, this APT source is "pinned" with a higher priority in /etc/apt/preferences.d/wikimedia.pref: 

Package: *
Pin: release o=Wikimedia
Pin-Priority: 1001

You can confirm Wikimedia's APT repository is taking preference now by running `apt-cache policy <package>`: 

$ apt-cache policy puppetmaster
puppetmaster:
  Installed: (none)
  Candidate: 4.8.2-5
  Version table:
     4.8.2-5 1001
        500 http://deb.debian.org/debian stretch/main amd64 Packages
       1001 http://apt.wikimedia.org/wikimedia stretch-wikimedia/main amd64 Packages

Security

The Wikimedia repository is signed with the Wikimedia Archive Automatic Signing Key <root@wikimedia.org>. This public key must be installed in APT's GPG keyring so it can verify packages successfully.

The key can be retrieved from /Stretch-Key (this key was introduced with stretch and also applies to later distros)

To install it, download the key to /etc/apt/trusted.gpg.d/wikimedia-apt-key.asc, or on older systems add it using `apt-key`: 

# Debian buster or later
$ wget -O /etc/apt/trusted.gpg.d/wikimedia-apt-key.asc "https://wikitech.wikimedia.org/w/index.php?title=APT_repository/Stretch-Key&action=raw"
# Debian Stretch or earlier
$ wget -O wikimedia-apt-key "https://wikitech.wikimedia.org/w/index.php?title=APT_repository/Stretch-Key&action=raw"
$ apt-key add wikimedia-apt-key
OK

You can also retrieve the key from a Cloud VPS host: ssh <host> apt-key export root@wikimedia.org

See also

  • pbuilder for building packages for different distribution versions and architectures
  • reprepro for additional information on how to use this tool
  • Wikimedia APT browser for a simple web package listing
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=APT_repository&oldid=2074724"