APT repository

From Wikitech
Jump to: navigation, search

Wikimedia maintains its own APT repository, under http://apt.wikimedia.org/wikimedia which lives on install1002. This repository contains Debian/Ubuntu packages modified for use by Wikimedia and Wikimedia specific native packages, that don't exist elsewhere.

Repository structure

Starting with stretch we are using a new repository structure:

  • main contains all the packages that we either create internally or packages that we backport or modify from Debian and which are useful fleet-wide (e.g. Icinga plugins, Cumin etc.)
  • thirdparty/hwraid contains binary-only drivers for hardware/RAID management/monitoring, these are only added on baremetal servers
  • All other repository components using the thirdparty/ prefix are synchronised from external repositories, e.g. thirdparty/cloudera, thirdparty/confluent, thirdparty/ci (Jenkins) and thirdparty/k8s (docker). A complete list can be found in our puppet config. Please do not add internally built packages to thirdparty.
  • Some packages should not be available fleet-wide, but rather be used in more specific scenarios. One example is component/ci, which provides co-installable PHP packages which are only necessary for CI. A complete list can be found in our puppet config.

For our older distros this split was not applied retroactively as it would have taken quite a bit of effort, so there are some differences:

  • For jessie and trusty internally built packages are spread out across main and backports. If you add new packages to jessie or trusty, simply always use main.
  • For jessie and trusty there's a general thirdparty component, not fully split into logical components (some already exist, e.g. thirdparty//cloudera). If you update an existing deb, update the respective thirdpary/foo components, otherwise put it in thirdparty (or even better migrate to stretch!)
  • Trusty also has universe and non-free, simply ignore this and use main.

Accessing the repository externally

For Wikimedia servers and Cloud VPS instances the repositories are automatically configured via puppet.

To use this repository from an external host, the following lines need to be present in /etc/apt/sources.list or /etc/apt/sources.d/wikimedia.list:

## Wikimedia APT repository
deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main experimental backports
deb-src http://apt.wikimedia.org/wikimedia jessie-wikimedia main universe

Use the right distribution (e.g. trusty-wikimedia) instead of jessie-wikimedia, depending on which Debian/Ubuntu version was installed.

A complete list is

deb http://apt.wikimedia.org/wikimedia trusty-wikimedia main experimental backports
deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main experimental backports
deb-src http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports
deb-src http://apt.wikimedia.org/wikimedia trusty-wikimedia main

Additionally, to make sure that the system prefers packages from this repository, and not packages from the origin distributions even if they have a higher version number, this APT source is "pinned" with a higher priority in /etc/apt/preferences:

Package: *
Pin: release o=Wikimedia
Pin-Priority: 1001

The Wikimedia repository is signed with the Wikimedia Archive Automatic Signing Key <root@wikimedia.org>. This public key must be installed in APT's GPG keyring for it to verify packages without nagging about it. This can be accomplished by installing the wikimedia-keyring package.

Alternative options:

  • get the key from /Key or for stretch /Stretch-Key
    $ wget -O wikimedia-apt-key "https://wikitech.wikimedia.org/w/index.php?title=APT_repository/Key&action=raw"
    $ apt-key add wikimedia-apt-key
    OK
    

Stretch:

  • $ wget -O wikimedia-apt-key "https://wikitech.wikimedia.org/w/index.php?title=APT_repository/Stretch-Key&action=raw"
    $ apt-key add wikimedia-apt-key
    OK
    
  • pull the key from a Cloud VPS host: ssh <host> apt-key export root@wikimedia.org, then add with apt-key add.

See also

  • pbuilder for building packages for different distribution versions and architectures
  • reprepro for additional information on how to use this tool