Debian packaging/Tutorial

This is a guide to git-based Debian packaging for WMF; it's about taking an upstream software package which is not currently in Debian and arranging for WMF CI to build Debian binary packages suitable for deployment at WMF. If the package is already available in Debian (or Ubuntu), fetch the Debianised source with dgit clone and follow Debian_packaging_with_dgit_and_CI. Similarly, if upstream already ship a debian/ directory, you may well be able to just start building from that.

This is not quite the same process as preparing a package for upload to Debian proper - we are primarily interested in a git-based workflow (and don't care about source packages, so can discard quilt-related complexity), and want to target a wikimedia-specific suite (e.g. bookworm-wikimedia). But we are still going to largely follow the process outlined in dgit-maint-merge(7); the model is that we have a branch of packaging-related changes that we merge with upstream release tags.

As an example, we are going to package unifdef, ignoring for the moment that there is a (slightly out-of-date) version of this software already in Debian.

Executive Summary / TL;DR

If you already know about Debian packaging, the relevant things you need to know/do differently to get your package build by WMF CI:

Use a non-quilt source format (e.g. 1.0), or at least arrange for the packaging branch(es) you want the CI to build to be patches-applied trees
Give your packaging branch(es) names that match the regex ^.*-wikimedia.*$ (e.g. packaging-wikimedia)
Specify the correct suite you want your package built for (e.g. bookworm-wikimedia) in debian/changelog
Set your CI/CD file to builddebs.yml@repos/sre/wmf-debci

This will arrange to build your package; to upload it to the WMF Apt repository see Debian_packaging#Upload to Wikimedia Repo

Minimum requirements

Assuming you will be following this guide on a Debian (or Ubuntu) machine, you will need at least the following packages installed: build-essential, devscripts, dh-make, git.

Getting The Source

Clone upstream, and name the created remote "upstream", then make our branch packaging-wikimedia off unifdef-2.11, the upstream release tag we want to work from.

 git clone -oupstream git://dotat.at/unifdef.git
 cd unifdef
 git checkout unifdef-2.11
 git checkout -b packaging-wikimedia

We call our branch packaging-wikimedia so that CI will know it's a branch to attempt to build from (it looks for branches matching the regexp .*-wikimedia.*$)

Initial Debianisation with `dh_make`

There are more specific tools for some languages (e.g. dh-make-golang), and use of dh_make is optional - you can create the necessary files under debian/ by yourself if you wish.

Debian source trees have a debian/ directory, which contains everything necessary to build Debian package(s) from the source. For the simplest of software (with the best-written upstream build systems), this can be as little as four files (debian/control, debian/copyright, debian/changelog, and debian/rules), but in practice you will often need a few more files to tell debhelper how to assemble binary packages from the upstream build process. dh_make is a tool that sets up a debian/ directory with some sensible defaults and placeholders for the files you're most likely to want to work with.

We've not bothered with an upstream tarball (we're relying on their release tags), but dh_make wants there to be one. But we can tell it to build one for us if we specify what package name and version our code relates to. We also specify our email address, and rather than a standard license (in which case dh_make will make a suitable debian/copyright for us) tell dh_make we have a custom license and where to find it (confusingly, relative to the new debian/ subdirectory). This is because the license for unifdef is quite complex.

 dh_make --createorig -p unifdef_2.11 -e mvernon@wikimedia.org -c custom --copyrightfile ../COPYING

dh_make will ask what sort of software this is (in our case, a single binary), and to confirm that its got the basics right:

 Type of package: (single, indep, library, python)
 [s/i/l/p]? s
 Maintainer Name     : Matthew Vernon
 Email-Address       : mvernon@wikimedia.org
 Date                : Tue, 10 Oct 2023 12:18:21 +0100
 Package Name        : unifdef
 Version             : 2.11
 License             : custom
 Package Type        : single
 Are the details correct? [Y/n/q] y

It has now created and populated the debian/ directory:

 $ ls debian
 changelog      manpage.sgml.ex  prerm.ex         source               watch.ex
 control        manpage.xml.ex   README.Debian    unifdef.cron.d.ex
 copyright      postinst.ex      README.source    unifdef.doc-base.ex
 manpage.1.ex   postrm.ex        rules            unifdef-docs.docs
 manpage.md.ex  preinst.ex       salsa-ci.yml.ex  upstream

Many of these files aren't going to be necessary for our very simple example, but for now commit the entire directory so we can later refer back to what dh_make did, if necessary - git add debian ; git commit -m "output of dh_make".

Now we turn to the key files we need to update for our new package - debian/source/format, debian/control, debian/copyright, debian/changelog, and debian/rules.

Debian Source Format - `debian/source/format`

dh_make has defaulted to "3.0 (quilt)" for the Debian source format. Quilt is essentially a way of storing revision control information inside a source package - and we don't want that, since it makes life more complicated, and we are instead storing our revision control information in git. So edit debian/source/format to instead contain 1.0 and commit the result.

 $ echo 1.0 >debian/source/format
 $ git add debian/source/format ; git commit -m "Set format to 1.0"
 [packaging-wikimedia 20eba40] Set format to 1.0
 1 file changed, 1 insertion(+), 1 deletion(-)

The Control File - `debian/control`

This file tells the package management system about the software you're packaging, and the source and binary packages built from it. Its structure and contents are defined in Debian policy.

For our purposes, we fill out the Section, Homepage, and Description fields, and delete the two VCS fields - in practice, you'd want to fill those out with the details of where on gitlab.wikimedia.org you were going to put the software. The result looks like this (remember to commit when done!):

 Source: unifdef
 Section: devel
 Priority: optional
 Maintainer: Matthew Vernon <mvernon@wikimedia.org>
 Rules-Requires-Root: no
 Build-Depends:
  debhelper-compat (= 13),
 Standards-Version: 4.6.2
 Homepage: https://dotat.at/prog/unifdef/
 
 Package: unifdef
 Architecture: any
 Depends:
  ${shlibs:Depends},
  ${misc:Depends},
 Description: selectively remove C preprocessor conditionals
  The unifdef utility selectively processes conditional C preprocessor
  #if and #ifdef directives. It removes from a file both the directives
  and the additional text that they delimit, while otherwise leaving
  the file alone.

Copyright information - `debian/copyright`

We have to be sure that the software we're distributing via our Apt repository is software that we're legally able to distribute (we do have an internal-only repository too, but that's out of scope here); and most Free Software requires a suitable copyright declaration to be included with the software. We do that by editing debian/copyright (which will end up in /usr/share/doc/packagename/copyright in our package). There is a machine-readable format for this file (which dh_make will part-fill-out for you if you specify a standard license) which is optional to use - for WMF purposes, just including the upstream copyright text verbatim will often be good enough (but do check that it's sufficiently detailed and clear).

Debian changelog - `debian/changelog`

The format of Debian changelogs are a little more strict than for changelogs in general, but dh_make has provided a plausible starter entry. If you use emacs, then the elpa-dpkg-dev-el package contains a useful mode for Debian changelogs. You should change the suite from UNRELEASED to the WMF suite you're wanting to build for (e.g. bookworm-wikimedia), and update the commit message. The result should look something like:

 unifdef (2.11-1) bookworm-wikimedia; urgency=medium
 
   * Package unifdef for WMF use
 
  -- Matthew Vernon <mvernon@wikimedia.org>  Tue, 10 Oct 2023 12:18:21 +0100

Getting the suite right is important, as WMF CI uses the suite in the changelog to pick the right image to build your package against.

Rules for package building - `debian/rules`

The debian/rules file produced by dh_make is a good starting point - do uncomment the export DH_VERBOSE = 1 line, as that will make fixing any build problems much easier. Unfortunately, the upstream build-system is not entirely well-behaved, so we have to make two additions to the rules file (after the existing dh $@ line):

 override_dh_auto_clean:
 	make clean
 	rm -f unifdef.txt
 
 override_dh_auto_install:
 	dh_auto_install -- prefix=/usr

But note that that the command lines must start with a tab, not spaces! What we have done here is override a couple of the standard debhelper commands; for more details on how this works (and more examples), see the dh(1) manual.

Tidying up

dh_make created a number of example and/or placeholder files that we don't actually need. Feel free to have a look at them to see what they do, but we now delete them:

 git rm debian/*.ex debian/unifdef-docs.docs debian/README.{Debian,source} debian/upstream/metadata.ex
 git commit -m "tidy up unnecessary examples from dh_make"

Source changes

Naturally, we might want to make changes to upstream source, too. That can be done like any other git operation - make changes, then commit. It's best to keep changes to upstream code in separate commits to changes to debian/, as that both makes sending fixes upstream easier, and makes merging new upstream versions more straightforward.

In our case, one of the tests fails (because CI builds run as root, and root can write to a mode 555 directory whilst a regular user cannot), so we want to remove the offending test:

 git rm tests/outeperm.{experr,expout,exprc,sh}
 git commit -m "Remove test that fails under CI builds (as root)"

Building

We now are in a position to (attempt to) build our new package. We can do this using CI (which is what we'll want to do in the end for production use), and/or build locally first (which can be useful if the build is not straightforward).

Building with CI

This is pretty easy (and described in more detail elsewhere on wikitech) - create a new empty repository (under repos/ since you need access to CI runners), and set the CI/CD file (found under Settings -> CI/CD -> General pipelines) to builddebs.yml@repos/sre/wmf-debci. For this example, I created a repository at repos/data_persistence/matthewvernon-demo-unifdef; gitlab itself then tells you how to upload your work (under "Push an existing Git repository"), although we don't need to rename an existing origin remote.

 git remote add origin git@gitlab.wikimedia.org:repos/data_persistence/matthewvernon-demo-unifdef.git
 git push --set-upstream origin --all
 git push --set-upstream origin --tags

If you now visit your gitlab project page, then under Build -> Pipelines, you should see a pipeline that is building your package. Assuming that works, the build artifact called build_ci_deb:archive will be a zip file containing a directory WMF_BUILD_DIR that contains the built packages, a .changes file, and a .buildinfo file, which you can inspect as described below.

Building Locally

If you want to build locally, make sure your git tree is clean (since afterwards you'll want to do git clean -df to tidy up your working tree again), then run

 dpkg-buildpackage -uc -b

Assuming that succeeds, your packages will be in ...

Inspecting the Result

You can use dpkg --info and dpkg --contents to inspect the .deb you've built, and lintian on the .changes file to look for packaging errors. Use lintian -i to get more information about any errors or warnings you're unsure about. The package built in this tutorial should look something like this:

 $ dpkg --contents WMF_BUILD_DIR/unifdef_2.11-1_amd64.deb 
 drwxr-xr-x root/root         0 2023-10-10 12:18 ./
 drwxr-xr-x root/root         0 2023-10-10 12:18 ./usr/
 drwxr-xr-x root/root         0 2023-10-10 12:18 ./usr/bin/
 -rwxr-xr-x root/root     43552 2023-10-10 12:18 ./usr/bin/unifdef
 -rwxr-xr-x root/root      2096 2023-10-10 12:18 ./usr/bin/unifdefall
 drwxr-xr-x root/root         0 2023-10-10 12:18 ./usr/share/
 drwxr-xr-x root/root         0 2023-10-10 12:18 ./usr/share/doc/
 drwxr-xr-x root/root         0 2023-10-10 12:18 ./usr/share/doc/unifdef/
 -rw-r--r-- root/root       154 2023-10-10 12:18 ./usr/share/doc/unifdef/changelog.Debian.gz
 -rw-r--r-- root/root      4278 2023-10-10 12:18 ./usr/share/doc/unifdef/copyright
 drwxr-xr-x root/root         0 2023-10-10 12:18 ./usr/share/man/
 drwxr-xr-x root/root         0 2023-10-10 12:18 ./usr/share/man/man1/
 -rw-r--r-- root/root      4641 2023-10-10 12:18 ./usr/share/man/man1/unifdef.1.gz
 lrwxrwxrwx root/root         0 2023-10-10 12:18 ./usr/share/man/man1/unifdefall.1.gz -> unifdef.1.gz
 
 $ lintian WMF_BUILD_DIR/unifdef_2.11-1_amd64.changes 
 E: unifdef changes: bad-distribution-in-changes-file bookworm-wikimedia
 W: unifdef: initial-upload-closes-no-bugs [usr/share/doc/unifdef/changelog.Debian.gz:1]

Note that both the lintian complaints are things we expect (and thus nothing to worry about) - we intentionally used the suite bookworm-wikimedia, and our upload to the WMF repo won't close any Debian bugs.

New Upstream Version

Often upstream releases new versions of software, and we want to update our package accordingly. Broadly, this is done by fetching the new upstream version and then merging that into our packaging branch. So far, we have been packaging version 2.11 of unifdef; but a newer version (2.22) is available. So we fetch (with --tags to make sure we get all upstream tags), and then merge the release tag we want onto our branch, before making a debian/changelog entry for our new version.

 git fetch --tags upstream
 git checkout packaging-wikimedia # this may well be a noop
 git merge unifdef-2.12
 dch -v2.12-1 New upstream release.
 dch -r # will spawn an editor for you to check the new changelog entry
 git add debian/changelog ; git commit -m "Changelog for 2.12-1"

The dilligent will want to check the upstream changes (with a snappy command such as git merge-tree $(git merge-base packaging-wikimedia unifdef-2.12) packaging-wikimedia unifdef-2.12 | less) before doing the merge, but we skip that for tutorial purposes. Once happy, you can git push and the CI will build the new version.

Executive Summary / TL;DR

Minimum requirements

Getting The Source

Initial Debianisation with dh_make

Debian Source Format - debian/source/format

The Control File - debian/control

Copyright information - debian/copyright

Debian changelog - debian/changelog

Rules for package building - debian/rules