Debian packaging/Tutorial
This is a guide to git-based Debian packaging for WMF; it's about taking an upstream software package which is not currently in Debian and arranging for WMF CI to build Debian binary packages suitable for deployment at WMF. If the package is already available in Debian (or Ubuntu), fetch the Debianised source with dgit clone and follow Debian_packaging_with_dgit_and_CI. Similarly, if upstream already ship a debian/ directory, you may well be able to just start building from that.
This is not quite the same process as preparing a package for upload to Debian proper - we are primarily interested in a git-based workflow (and don't care about source packages, so can discard quilt-related complexity), and want to target a wikimedia-specific suite (e.g. bookworm-wikimedia). But we are still going to largely follow the process outlined in dgit-maint-merge(7); the model is that we have a branch of packaging-related changes that we merge with upstream release tags.
As an example, we are going to package unifdef, ignoring for the moment that there is a (slightly out-of-date) version of this software already in Debian.
Executive Summary / TL;DR
If you already know about Debian packaging, the relevant things you need to know/do differently to get your package build by WMF CI:
- Use a non-quilt source format (e.g. 1.0), or at least arrange for the packaging branch(es) you want the CI to build to be patches-applied trees
- Give your packaging branch(es) names that match the regex
^.*-wikimedia.*$(e.g. packaging-wikimedia) - Specify the correct suite you want your package built for (e.g.
bookworm-wikimedia) indebian/changelog - Set your CI/CD file to
builddebs.yml@repos/sre/wmf-debci
This will arrange to build your package; to upload it to the WMF Apt repository see Debian_packaging#Upload to Wikimedia Repo
Minimum requirements
Assuming you will be following this guide on a Debian (or Ubuntu) machine, you will need at least the following packages installed: build-essential, devscripts, dh-make, git.
Getting The Source
Clone upstream, and name the created remote "upstream", then make our branch packaging-wikimedia off unifdef-2.11, the upstream release tag we want to work from.
git clone -oupstream git://dotat.at/unifdef.git cd unifdef git checkout unifdef-2.11 git checkout -b packaging-wikimedia
We call our branch packaging-wikimedia so that CI will know it's a branch to attempt to build from (it looks for branches matching the regexp .*-wikimedia.*$)
Initial Debianisation with dh_make
Debian source trees have a debian/ directory, which contains everything necessary to build Debian package(s) from the source. For the simplest of software (with the best-written upstream build systems), this can be as little as four files (debian/control, debian/copyright, debian/changelog, and debian/rules), but in practice you will often need a few more files to tell debhelper how to assemble binary packages from the upstream build process. dh_make is a tool that sets up a debian/ directory with some sensible defaults and placeholders for the files you're most likely to want to work with.
We've not bothered with an upstream tarball (we're relying on their release tags), but dh_make wants there to be one. But we can tell it to build one for us if we specify what package name and version our code relates to. We also specify our email address, and rather than a standard license (in which case dh_make will make a suitable debian/copyright for us) tell dh_make we have a custom license and where to find it (confusingly, relative to the new debian/ subdirectory). This is because the license for unifdef is quite complex.
dh_make --createorig -p unifdef_2.11 -e mvernon@wikimedia.org -c custom --copyrightfile ../COPYING
dh_make will ask what sort of software this is (in our case, a single binary), and to confirm that its got the basics right:
Type of package: (single, indep, library, python) [s/i/l/p]? s Maintainer Name : Matthew Vernon Email-Address : mvernon@wikimedia.org Date : Tue, 10 Oct 2023 12:18:21 +0100 Package Name : unifdef Version : 2.11 License : custom Package Type : single Are the details correct? [Y/n/q] y
It has now created and populated the debian/ directory:
$ ls debian changelog manpage.sgml.ex prerm.ex source watch.ex control manpage.xml.ex README.Debian unifdef.cron.d.ex copyright postinst.ex README.source unifdef.doc-base.ex manpage.1.ex postrm.ex rules unifdef-docs.docs manpage.md.ex preinst.ex salsa-ci.yml.ex upstream
Many of these files aren't going to be necessary for our very simple example, but for now commit the entire directory so we can later refer back to what dh_make did, if necessary - git add debian ; git commit -m "output of dh_make".
Now we turn to the key files we need to update for our new package - debian/source/format, debian/control, debian/copyright, debian/changelog, and debian/rules.
Debian Source Format - debian/source/format
dh_make has defaulted to "3.0 (quilt)" for the Debian source format. Quilt is essentially a way of storing revision control information inside a source package - and we don't want that, since it makes life more complicated, and we are instead storing our revision control information in git. So edit debian/source/format to instead contain 1.0 and commit the result.
$ echo 1.0 >debian/source/format $ git add debian/source/format ; git commit -m "Set format to 1.0" [packaging-wikimedia 20eba40] Set format to 1.0 1 file changed, 1 insertion(+), 1 deletion(-)
The Control File - debian/control
This file tells the package management system about the software you're packaging, and the source and binary packages built from it. Its structure and contents are defined in Debian policy.
For our purposes, we fill out the Section, Homepage, and Description fields, and delete the two VCS fields - in practice, you'd want to fill those out with the details of where on gitlab.wikimedia.org you were going to put the software. The result looks like this (remember to commit when done!):
Source: unifdef Section: devel Priority: optional Maintainer: Matthew Vernon <mvernon@wikimedia.org> Rules-Requires-Root: no Build-Depends: debhelper-compat (= 13), Standards-Version: 4.6.2 Homepage: https://dotat.at/prog/unifdef/ Package: unifdef Architecture: any Depends: ${shlibs:Depends}, ${misc:Depends}, Description: selectively remove C preprocessor conditionals The unifdef utility selectively processes conditional C preprocessor #if and #ifdef directives. It removes from a file both the directives and the additional text that they delimit, while otherwise leaving the file alone.
Copyright information - debian/copyright
We have to be sure that the software we're distributing via our Apt repository is software that we're legally able to distribute (we do have an internal-only repository too, but that's out of scope here); and most Free Software requires a suitable copyright declaration to be included with the software. We do that by editing debian/copyright (which will end up in /usr/share/doc/packagename/copyright in our package). There is a machine-readable format for this file (which dh_make will part-fill-out for you if you specify a standard license) which is optional to use - for WMF purposes, just including the upstream copyright text verbatim will often be good enough (but do check that it's sufficiently detailed and clear).
Debian changelog - debian/changelog
The format of Debian changelogs are a little more strict than for changelogs in general, but dh_make has provided a plausible starter entry. If you use emacs, then the elpa-dpkg-dev-el package contains a useful mode for Debian changelogs. You should change the suite from UNRELEASED to the WMF suite you're wanting to build for (e.g. bookworm-wikimedia), and update the commit message. The result should look something like:
unifdef (2.11-1) bookworm-wikimedia; urgency=medium * Package unifdef for WMF use -- Matthew Vernon <mvernon@wikimedia.org> Tue, 10 Oct 2023 12:18:21 +0100
Getting the suite right is important, as WMF CI uses the suite in the changelog to pick the right image to build your package against.
Rules for package building - debian/rules
The debian/rules file produced by dh_make is a good starting point - do uncomment the export DH_VERBOSE = 1 line, as that will make fixing any build problems much easier. Unfortunately, the upstream build-system is not entirely well-behaved, so we have to make two additions to the rules file (after the existing dh $@ line):
override_dh_auto_clean: make clean rm -f unifdef.txt override_dh_auto_install: dh_auto_install -- prefix=/usr
But note that that the command lines must start with a tab, not spaces! What we have done here is override a couple of the standard debhelper commands; for more details on how this works (and more examples), see the dh(1) manual.
Tidying up
dh_make created a number of example and/or placeholder files that we don't actually need. Feel free to have a look at them to see what they do, but we now delete them:
git rm debian/*.ex debian/unifdef-docs.docs debian/README.{Debian,source} debian/upstream/metadata.ex
git commit -m "tidy up unnecessary examples from dh_make"
Source changes
Naturally, we might want to make changes to upstream source, too. That can be done like any other git operation - make changes, then commit. It's best to keep changes to upstream code in separate commits to changes to debian/, as that both makes sending fixes upstream easier, and makes merging new upstream versions more straightforward.
In our case, one of the tests fails (because CI builds run as root, and root can write to a mode 555 directory whilst a regular user cannot), so we want to remove the offending test:
git rm tests/outeperm.{experr,expout,exprc,sh}
git commit -m "Remove test that fails under CI builds (as root)"
Building
We now are in a position to (attempt to) build our new package. We can do this using CI (which is what we'll want to do in the end for production use), and/or build locally first (which can be useful if the build is not straightforward).
Building with CI
This is pretty easy (and described in more detail elsewhere on wikitech) - create a new empty repository (under repos/ since you need access to CI runners), and set the CI/CD file (found under Settings -> CI/CD -> General pipelines) to builddebs.yml@repos/sre/wmf-debci. For this example, I created a repository at repos/data_persistence/matthewvernon-demo-unifdef; gitlab itself then tells you how to upload your work (under "Push an existing Git repository"), although we don't need to rename an existing origin remote.
git remote add origin git@gitlab.wikimedia.org:repos/data_persistence/matthewvernon-demo-unifdef.git git push --set-upstream origin --all git push --set-upstream origin --tags
If you now visit your gitlab project page, then under Build -> Pipelines, you should see a pipeline that is building your package. Assuming that works, the build artifact called build_ci_deb:archive will be a zip file containing a directory WMF_BUILD_DIR that contains the built packages, a .changes file, and a .buildinfo file, which you can inspect as described below.
Building Locally
If you want to build locally, make sure your git tree is clean (since afterwards you'll want to do git clean -df to tidy up your working tree again), then run
dpkg-buildpackage -uc -b
Assuming that succeeds, your packages will be in ...
Inspecting the Result
You can use dpkg --info and dpkg --contents to inspect the .deb you've built, and lintian on the .changes file to look for packaging errors. Use lintian -i to get more information about any errors or warnings you're unsure about. The package built in this tutorial should look something like this:
$ dpkg --contents WMF_BUILD_DIR/unifdef_2.11-1_amd64.deb drwxr-xr-x root/root 0 2023-10-10 12:18 ./ drwxr-xr-x root/root 0 2023-10-10 12:18 ./usr/ drwxr-xr-x root/root 0 2023-10-10 12:18 ./usr/bin/ -rwxr-xr-x root/root 43552 2023-10-10 12:18 ./usr/bin/unifdef -rwxr-xr-x root/root 2096 2023-10-10 12:18 ./usr/bin/unifdefall drwxr-xr-x root/root 0 2023-10-10 12:18 ./usr/share/ drwxr-xr-x root/root 0 2023-10-10 12:18 ./usr/share/doc/ drwxr-xr-x root/root 0 2023-10-10 12:18 ./usr/share/doc/unifdef/ -rw-r--r-- root/root 154 2023-10-10 12:18 ./usr/share/doc/unifdef/changelog.Debian.gz -rw-r--r-- root/root 4278 2023-10-10 12:18 ./usr/share/doc/unifdef/copyright drwxr-xr-x root/root 0 2023-10-10 12:18 ./usr/share/man/ drwxr-xr-x root/root 0 2023-10-10 12:18 ./usr/share/man/man1/ -rw-r--r-- root/root 4641 2023-10-10 12:18 ./usr/share/man/man1/unifdef.1.gz lrwxrwxrwx root/root 0 2023-10-10 12:18 ./usr/share/man/man1/unifdefall.1.gz -> unifdef.1.gz $ lintian WMF_BUILD_DIR/unifdef_2.11-1_amd64.changes E: unifdef changes: bad-distribution-in-changes-file bookworm-wikimedia W: unifdef: initial-upload-closes-no-bugs [usr/share/doc/unifdef/changelog.Debian.gz:1]
Note that both the lintian complaints are things we expect (and thus nothing to worry about) - we intentionally used the suite bookworm-wikimedia, and our upload to the WMF repo won't close any Debian bugs.
New Upstream Version
Often upstream releases new versions of software, and we want to update our package accordingly. Broadly, this is done by fetching the new upstream version and then merging that into our packaging branch. So far, we have been packaging version 2.11 of unifdef; but a newer version (2.22) is available. So we fetch (with --tags to make sure we get all upstream tags), and then merge the release tag we want onto our branch, before making a debian/changelog entry for our new version.
git fetch --tags upstream git checkout packaging-wikimedia # this may well be a noop git merge unifdef-2.12 dch -v2.12-1 New upstream release. dch -r # will spawn an editor for you to check the new changelog entry git add debian/changelog ; git commit -m "Changelog for 2.12-1"
The dilligent will want to check the upstream changes (with a snappy command such as git merge-tree $(git merge-base packaging-wikimedia unifdef-2.12) packaging-wikimedia unifdef-2.12 | less) before doing the merge, but we skip that for tutorial purposes. Once happy, you can git push and the CI will build the new version.
Further Reading
This tutorial has, by design, picked a simple piece of software with straightforward packaging requirements. If your software packaging is more complex, it will be worth reading the debhelper manual which describes how to tweak all sorts of aspects of package building (it has utilities to install most sorts of things found in packages, usually configured by files in debian/) as well as linking to further documentation of how it (and dh) works.
If your upstream releases software as tarballs (or you want to package commits which upstream hasn't tagged), then the dgit-maint-merge manual has useful pointers into how to modify the approach described here. If you find yourself maintaining large and complex changes to upstream software which upstream aren't going to incorporate (this is not a good situation to be in!), then you might find dgit-maint-debrebase useful, but this is not a simple approach.
If you want to read (much) more about how Debian packages should be laid out, see the Debian policy manual.