APT repository

From Wikitech
Jump to: navigation, search

Wikimedia maintains its own APT repository, under http://apt.wikimedia.org/wikimedia which lives on install1002. This repository contains Debian/Ubuntu packages modified for use by Wikimedia, and Wikimedia specific native packages, that don't exist elsewhere.

Using

To use this repository, the following lines need to be present in /etc/apt/sources.list or /etc/apt/sources.d/wikimedia.list:

## Wikimedia APT repository
deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main experimental backports
deb-src http://apt.wikimedia.org/wikimedia jessie-wikimedia main universe

Use the right distribution (e.g. trusty-wikimedia) instead of jessie-wikimedia, depending on which Debian/Ubuntu version was installed.

A complete list is

deb http://apt.wikimedia.org/wikimedia precise-wikimedia main experimental backports
deb http://apt.wikimedia.org/wikimedia trusty-wikimedia main experimental backports
deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main experimental backports
deb-src http://apt.wikimedia.org/wikimedia precise-wikimedia main
deb-src http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports
deb-src http://apt.wikimedia.org/wikimedia trusty-wikimedia main

Additionally, to make sure that the system prefers packages from this repository, and not packages from the origin distributions even if they have a higher version number, this APT source is "pinned" with a higher priority in /etc/apt/preferences:

Package: *
Pin: release o=Wikimedia
Pin-Priority: 1001

The Wikimedia repository is signed with the Wikimedia Archive Automatic Signing Key <root@wikimedia.org>. This public key must be installed in APT's GPG keyring for it to verify packages without nagging about it. This can be accomplished by installing the wikimedia-keyring package.

Alternative options:

  • get the key from /Key or for stretch /Stretch-Key
    $ wget -O wikimedia-apt-key "https://wikitech.wikimedia.org/w/index.php?title=APT_repository/Key&action=raw"
    $ apt-key add wikimedia-apt-key
    OK
    

Stretch:

  • $ wget -O wikimedia-apt-key "https://wikitech.wikimedia.org/w/index.php?title=APT_repository/Stretch-Key&action=raw"
    $ apt-key add wikimedia-apt-key
    OK
    
  • pull the key from a labs host: ssh <host> apt-key export root@wikimedia.org, then add with apt-key add.

An automated installation of a Wikimedia server should take care of all of the above.

Old information as of 2012. See Reprepro

Previously, we were using one big package pool for all distribution versions we had installed, i.e., packages in hardy-wikimedia and lucid-wikimedia were the same. However, this gives all kinds of version incompatibilities, so we have now split the repository per distribution, and we are now managing the repository using a nice tool called reprepro.

It's important that we always specify the right distribution (hardy-wikimedia or lucid-wikimedia) for which the package is built in the package's Changelog (debian/changelog). The package should also be put under the right component, see below for details.


Updating the repository

To add packages to the Wikimedia repository, so they can be deployed on the servers, take the following steps.

Commit the debian/ directory of the package to Subversion, under:

svn.wikimedia.org/svnroot/mediawiki/trunk/debs/package basename/debian/

Make sure that the package is built on the relevant architectures (these days, only amd64). If you need to build a special package for each distribution version (as you likely will need to for at least binary packages), you should add the distribution name to the package version number. For example:

Package

puppet_2.6.1-0ubuntu1

becomes:

puppet_2.6.1-0ubuntu1~hardy1

and

puppet_2.6.1-0ubuntu1~lucid1

Be aware that a suffix starting with ~ sorts as a lower version than the original package without that suffix!

Upload all the package files, including the .changes file, to directory /srv/wikimedia/incoming/ on brewster. You can use the convenient tool dupload for this, on hosts on which it has been setup (e.g. ragweed):

$ dupload path-to-.changes-file

Then, on brewster, run:

# reprepro -C component-name include wikimedia-distribution-name path-to-.changes-file

For example:

# reprepro -C main include hardy-wikimedia php5-apc_3.1.3p1-1wm1_amd64.changes

If any previous version of the package was present in the repository, it will be removed completely by reprepro.

For component, use the following as a guide:

main 
for Wikimedia native packages, as well as Debian/Ubuntu packages that have had source-modifications
universe 
for existing Debian/Ubuntu packages that just have been recompiled/backported for the given distribution.

For distribution, use the distribution that the package has been compiled for, and under. Usually, any given compiled package should be for one distribution only, e.g. hardy-wikimedia OR lucid-wikimedia. This should match the field in the package's Changelog. Only in special circumstances a given package can be used under multiple distribution versions, .e.g. if the package contains only scripts (no binaries). Only do this if you've tested it and are sure this is working. In this case, include the package in the oldest distribution version, and then copy the package references using reprepro copy.

Under some circumstances (when you don't have a .changes file) you can use different reprepro commands, such as reprepro includedsc and reprepro includedeb. See reprepro for additional details.

The package should now be available for deployment/installation on the target hosts, after they have done a apt-get update, of course.

Setup of a Wikimedia repository for a new Ubuntu release

Edit /srv/wikimedia/conf/distributions, adding a block for the new distribution that matches the previous ones. For example:

Origin: Wikimedia
Label: Wikimedia
Suite: lucid-wikimedia
Codename: lucid-wikimedia
Version: 10.04
Architectures: source amd64
Components: main universe
UDebComponents: main
Description: Wikimedia specific packages for Ubuntu Lucid
SignWith: default
DebOverride: deb-override

New packages can now be imported for this distribution as normally.

See also

  • pbuilder for building packages for different distribution versions and architectures
  • reprepro for additional information on how to use this tool