Debian Packaging
These are runbooks related to Debian packaging for Wikimedia Foundation production.
Rebuilding a package
Process to repackage a Debian package and push it to wmf apt repo.
Example
To repackage python-libthumbor
:
- Login to
build2001.codfw.wmnet
- Run the following to download
python-libthumbor
's source package:$ apt-get source libthumbor=1.3.2-0+wmf1
- We should now have 3 files and 1 directory in our working directory:
- libthumbor_1.3.2.orig.tar.gz
- Source code of the upstream authors
- libthumbor_1.3.2-0+wmf1.dsc
- Meta file describing which files are around
- libthumbor_1.3.2-0+wmf1.debian.tar.xz
- Debian-specific changes
- libthumbor-1.3.2
- Extracted source package
- Note the Debian-specific files in the
libthumbor-1.3.2/debian
directory. For more information about these files, consult Debian's manual detailing Required files under the debian directory and Other files under the debian directory. - Navigate to the
libthumbor-1.3.2
directory and usedch
to add a new changelog entry to the top of thedebian/changelog
:$ cd libthumbor-1.3.2 $ dch -i
- Add the new version number, making sure that it's higher than the previous version.
- Change
UNRELEASED
toVERSION_CODENAME-wikimedia
whereVERSION_CODENAME
is the targeted Debian version. - Fill in a clear and accurate list of what changes have occurred since the last entry.
- An example entry might look like:
libthumbor (1.3.2-0+wmf1+stretch1) stretch-wikimedia; urgency=low * Rebuild for stretch -- Effie Mouzeli <effie@wikimedia.org> Tue, 20 Nov 2018 13:42:24 +0000 libthumbor (1.3.2-0+wmf1) unstable; urgency=low * Initial release. (Closes: #783577) -- Gilles Dubuc <gilles@wikimedia.org> Thu, 19 May 2016 15:30:16 +0000
- When we rebuild for a new distro we use ~, not + like in the example in that page, the difference being 1.5-1~something is less than 1.5-1 so if 1.5-1 ever appears on official buster, it will supersede our package.
- Submit a patch to Gerrit and wait for approval before merging.
- Build:
- To use packages from the
VERSION_CODENAME
repo runDIST=VERSION_CODENAME pdebuild
- To include packages from the
VERSION_CODENAME-wikimedia
repo, runDIST=VERSION_CODENAME-wikimedia pdebuild
- If building for a new Debian version do something like
DIST=VERSION_CODENAME WIKIMEDIA=yes pdebuild --debbuildopts -sa
- To use packages from the
- Find the build results in
/var/cache/pbuilder/result/VERSION_CODENAME-amd64/
:- libthumbor_1.3.2-0+wmf1+
VERSION_CODENAME
1_amd64.buildinfo - libthumbor_1.3.2-0+wmf1+
VERSION_CODENAME
1_amd64.changes - libthumbor_1.3.2-0+wmf1+
VERSION_CODENAME
1.debian.tar.xz - libthumbor_1.3.2-0+wmf1+
VERSION_CODENAME
1.dsc - python-libthumbor_1.3.2-0+wmf1+
VERSION_CODENAME
1_all.deb
- libthumbor_1.3.2-0+wmf1+
- Upload your artifacts to the Wikimedia repository.
Download source packages from snapshot.debian.org
If you are looking to build a different version than the one provided by a source package, you can have a look at Debian snapshots. Everything you need will be downloaded for you by pointing dget
to the relevant .dsc
file:
$ dget https://snapshot.debian.org/archive/debian-debug/20180826T210249Z/pool/main/libr/librsvg/librsvg_2.40.20-3.dsc
Package-specific deviations
- If we are building a new package where we are the owners, like python-thumbor-wikimedia, we should use
native
instead ofquilt
indebian/source/format
- If we are to use packages from eg php72 component, this can be done by using
PHP72=yes
env var before building.
Test with piuparts
Can be useful sometime to test how the just-build package behaves in terms of installing/upgrading/reinstalling. For this task the piuparts
can help us.
piuparts
is currently not installed on any buildhost. Consider installing locally, eg. on a VM on your computerOnce we have built the package we can test with:
# piuparts -d <distribution> <package>
Eg.
# piuparts -d bookworm varnishkafka_1.1.0-3_amd64.deb
If we want to test all the packages built from a single source package (ex. because there are some dependencies between them) we can pass directly the .changes
file:
# piuparts -d bookworm varnishkafka_1.1.0-3_amd64.changes
piuparts
will try to install/upgrade/purge all binary packages listed in the .changes
file.
If we want to test a single package against multiple distributions we can use:
# piuparts -d bullseye -d bookworm varnishkafka_1.1.0-3_amd64.changes
If we want to include an extra repository (eg. for getting dependencies needed by our newly-built package) we can use something like:
# piuparts -d bookworm --extra-repo='deb [ trusted=yes ] http://apt.wikimedia.org/wikimedia bookworm-wikimedia main' varnishkafka_1.1.0-3_amd64.deb
Upload to Wikimedia Repo
1. Login to apt1001.wikimedia.org
, create a directory and copy all files from build2001
$ ssh apt1001.wikimedia.org $ mkdir libthumbor; cd libthumbor $ rsync rsync://build2001.codfw.wmnet/pbuilder-result/stretch-amd64/*thum* .
2. Import!
# reprepro -C main include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes
Note: We split the archives sometimes, i.e. we add packages to components instead of main
3. Log your uploads in #wikimedia-operations on IRC using !log
Upload to a component directory
Sometimes we build packages for specific purposes thus we don't want them to be available under the main
component.
To create a new component, you simply need to prepare a patch for puppet/modules/aptrepo/files/distributions-wikimedia
. After the component is available on install*
servers, you can upload to it by running:
# reprepro -C component/thumbor include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes
Note: if you get an error like Error opening config file './conf/distributions': No such file or directory(2)
, then you forgot to do sudo -H bash
Create a Debian patch
Sometimes we might wish to add a fix to the software we are packaging, not included in the original source.
- Check if your source package is using quilt, Debian's implementation of patch series tracking:
$ cat debian/source/format 3.0 (quilt)
- Make the edits you wish to do for your fix and run the following command:
$ dpkg-source --commit librsvg-2.40.18/tests/crash.c Enter the desired patch name: something-with-sense.patch
- Name your patch and use the spawned editor to make a description, for instance:
Description: add -exact in webp conversion The -exact option was introduced in webp 0.5, which is only available on Stretch. python-thumbor-wikimedia (2.2-1+deb9u1) stretch-wikimedia; urgency=low * Repackaging for stretch Author: Effie Mouzeli <effie@wikimedia.org> ---
- Your patch will be saved under
debian/patches/foo.patch
. Build it in the usual way- If you are uploading a package to a
-wikimedia
repository that is a patched version of the version originally in Debian proper, you should also pass--debbuildopts -sa
to pdebuild. This will force it to include the.orig.tar.gz
containing the source code as provided by the original developer.
Browse current package source
To review the active patches we have applied to a package, there are two approaches one can take: standalone using dget
, or through apt source
. The former has the benefit of not needing to modify any /etc/apt/
configuration in the environment where you do this. The latter has the benefit of using the same tooling and logic as installing a package would (and thus involve changing /etc/ files).
Standalone using dget
- Find the latest
dsc
file for the package from https://apt.wikimedia.org/wikimedia/pool/. For example, kafkacat from the main component is under https://apt.wikimedia.org/wikimedia/pool/main/k/kafkacat/. Our custom component/php72 packages are under https://apt.wikimedia.org/wikimedia/pool/component/php72/p/php7.2/ - Ensure devscripts is installed, which provides the
dget
command. - Move to a temporary directory and run
dget -u https://apt.wikimedia.org/wikimedia/.../example-1.2.3-wmf1.dsc
- The package source is now in
example-1.2.3-wmf1/
, with a familiar "debian" subdirectory containing "debian/patches", etc.
Through apt
- Ensure
/etc/apt/sources.list
also includes the deb-src variant of the relevant sources.list lines. For example, when using component/php72 add the last line as well:
deb http://mirrors.wikimedia.org/debian/ buster main deb http://security.debian.org/ buster/updates main deb http://mirrors.wikimedia.org/debian/ buster-updates main deb http://mirrors.wikimedia.org/debian/ buster-backports main contrib deb http://apt.wikimedia.org/wikimedia buster-wikimedia main deb http://apt.wikimedia.org/wikimedia buster-wikimedia component/php72 deb-src http://apt.wikimedia.org/wikimedia buster-wikimedia component/php72
- Move to a temporary directory and run
apt source <pgk-name>
, e.g.apt source php7.2
. This will use the same resolution logic as forapt-get install
. - The package source is now in
<pgk-name>-1.2.3-wmf1/
, with a familiar "debian" subdirectory containing "debian/patches", etc.
See also
If a package is unchanged from upstream, then you can also browse the source online via Debian Salsa and its GitLab interface. For example:
https://salsa.debian.org/php-team/php/-/tree/debian/main/7.2/debian/patches
Troubleshooting
Dependency issues with debhelper when backporting
Sometimes the source package you are backporting depends upon a newer version of debhelper
than is available in the target distribution:
pbuilder-satisfydepends-dummy : Depends: debhelper (>= 9.20160709) but 9.20150101+deb8u2 is to be installed.
Adding BACKPORTS=yes
to the beginning of your pdebuild
invocation will often fix this.