Wikimedia uses Debian for its infrastructure. Libraries, modules and software are usually provided as a Debian package which is then uploaded to our APT repository. Packages are built by CI from git repositories, which simplifies aspects of the packaging workflow (e.g. we no longer need to wrestle with quilt or produce source packages), and enables automatic tracking of updates from Debian.
If you are rebuilding, backporting, or otherwise modifying existing Debian packages, then the workflow is described in Debian_packaging_with_dgit_and_CI. If you already know about Debian packaging, and want to know how to package something new and have it built by WMF CI, there is a short summary of what you need. Alternatively, there is a tutorial on creating new Debian packages for WMF use using dh (which is Debian tooling to simplify the creation of Debian packages).
Rebuilding a package
Process to repackage a Debian package and push it to wmf apt repo. Soon we will have an automatically-available staging repository which you can use, but in the mean time, build your package with CI, and then upload the resulting package to the WMF Apt repository:
- Follow the process described in Debian_packaging_with_dgit_and_CI to build your package.
- Upload your artifacts to the Wikimedia repository.
Test with piuparts
Can be useful sometime to test how the just-build package behaves in terms of installing/upgrading/reinstalling. For this task the
piuparts can help us.
Once we have built the package we can test with:
# piuparts -d <distribution> <package>
# piuparts -d bookworm varnishkafka_1.1.0-3_amd64.deb
If we want to test all the packages built from a single source package (ex. because there are some dependencies between them) we can pass directly the
# piuparts -d bookworm varnishkafka_1.1.0-3_amd64.changes
piuparts will try to install/upgrade/purge all binary packages listed in the
If we want to test a single package against multiple distributions we can use:
# piuparts -d bullseye -d bookworm varnishkafka_1.1.0-3_amd64.changes
If we want to include an extra repository (eg. for getting dependencies needed by our newly-built package) we can use something like:
# piuparts -d bookworm --extra-repo='deb [ trusted=yes ] http://apt.wikimedia.org/wikimedia bookworm-wikimedia main' varnishkafka_1.1.0-3_amd64.deb
Upload to Wikimedia Repo
1. Login to
apt1001.wikimedia.org, create a directory and copy the build artifact (called build_ci_deb:archive in the gitlab UI) there. When you
unzip the build artifact there will be a directory called
WMF_BUILD_DIR, which will contain a number of files, including a
.changes file, which you pass to
$ unzip bookworm.zip $ sudo -i reprepro -C main include bullseye-wikimedia WMF_BUILD_DIR/packagename_version_amd64.changes
2. Log your uploads in #wikimedia-operations on IRC using
Upload to a component directory
Sometimes we build packages for specific purposes thus we don't want them to be available under the
To create a new component, you simply need to prepare a patch for
puppet/modules/aptrepo/files/distributions-wikimedia. After the component is available on
install* servers, you can upload to it by running:
# reprepro -C component/thumbor include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes
Note: if you get an error like
Error opening config file './conf/distributions': No such file or directory(2), then you forgot to do
sudo -H bash
Create a Debian patch
Sometimes we might wish to add a fix to the software we are packaging, not included in the original source. Since we are using git to store our source code and track changes to it, we don't need to bother with quilt and source packages ourselves. Instead, we can simply make changes on our packaging git branch as if it were a normal git repository, as described in Debian_packaging_with_dgit_and_CI#Making_Changes.
Browse current package source
Our Debian packages are maintained in git repositories, so to review the code for a package,
git clone the relevant repository from WMF gitlab. The branch naming convention is described elsewhere, but briefly, branches called
dgit/bookworm track the relevant Debian release, and branches called
wikimedia-bookworm are packaging branches.
For older packages that were maintained using a source-package-based workflow (rather than a git-based one), you need to instead use
dget to retrieve the source:
Retrieving old-style source packages using
- Find the latest
dscfile for the package from https://apt.wikimedia.org/wikimedia/pool/. For example, kafkacat from the main component is under https://apt.wikimedia.org/wikimedia/pool/main/k/kafkacat/. Our custom component/php72 packages are under https://apt.wikimedia.org/wikimedia/pool/component/php72/p/php7.2/
- Ensure devscripts is installed, which provides the
- Move to a temporary directory and run
dget -u https://apt.wikimedia.org/wikimedia/.../example-1.2.3-wmf1.dsc
- The package source is now in
example-1.2.3-wmf1/, with a "debian" subdirectory containing "debian/patches", etc. NB, these patches will not have been applied to the source code.
Getting source from Debian
Always use dgit clone to fetch sources from Debian.
Dependency issues with debhelper when backporting
Sometimes the source package you are backporting depends upon a newer version of
debhelper than is available in the target distribution:
pbuilder-satisfydepends-dummy : Depends: debhelper (>= 9.20160709) but 9.20150101+deb8u2 is to be installed.
BACKPORTS=yes to the beginning of your
pdebuild invocation will often fix this.