Jump to content

Debian packaging

From Wikitech
(Redirected from Package management)

Wikimedia uses Debian for its infrastructure. Libraries, modules and software are usually provided as a Debian package which is then uploaded to our APT repository. Packages are built by CI from git repositories, which simplifies aspects of the packaging workflow (e.g. we no longer need to wrestle with quilt or produce source packages), and enables automatic tracking of updates from Debian.

If you are rebuilding, backporting, or otherwise modifying existing Debian packages, then the workflow is described in Debian_packaging_with_dgit_and_CI. If you already know about Debian packaging, and want to know how to package something new and have it built by WMF CI, there is a short summary of what you need. Alternatively, there is a tutorial on creating new Debian packages for WMF use using dh (which is Debian tooling to simplify the creation of Debian packages).

Rebuilding a package

Packaging a deb and uploading to the WMF apt repo is soon going to be an automated process, with packages built by CI and then uploaded to a staging repository from which they can be promoted. While the process is not yet completed, you can already use the gitlab CI to build your package automatically.

We should distinguish two main cases:

  • When you're repackaging a package from debian, adding patches or tweaking the packaging, you should follow the process described in Debian_packaging_with_dgit_and_CI to build your package
  • When you're packaging a software we develop (and intend it only for local use), it's easier to follow the process for packaging your software as deb

Once the package is built you can upload your artifacts to the Wikimedia repository.

Test with piuparts

Can be useful sometime to test how the just-build package behaves in terms of installing/upgrading/reinstalling. For this task the piuparts can help us.

Once we have built the package we can test with: 

# piuparts -d <distribution> <package>

Eg. 

# piuparts -d bookworm varnishkafka_1.1.0-3_amd64.deb

If we want to test all the packages built from a single source package (ex. because there are some dependencies between them) we can pass directly the .changes file: 

# piuparts -d bookworm varnishkafka_1.1.0-3_amd64.changes

piuparts will try to install/upgrade/purge all binary packages listed in the .changes file.

If we want to test a single package against multiple distributions we can use: 

# piuparts -d bullseye -d bookworm varnishkafka_1.1.0-3_amd64.changes

If we want to include an extra repository (eg. for getting dependencies needed by our newly-built package) we can use something like: 

# piuparts -d bookworm --extra-repo='deb [ trusted=yes ] http://apt.wikimedia.org/wikimedia bookworm-wikimedia main' varnishkafka_1.1.0-3_amd64.deb

Upload to Wikimedia Repo

1. Login to apt1002.wikimedia.org, create a directory and copy the build artifact (called build_ci_deb:archive in the gitlab UI) there. When you unzip the build artifact there will be a directory called WMF_BUILD_DIR, which will contain a number of files, including a .changes file, which you pass to reprepro 

$ unzip bookworm.zip
$ sudo -i reprepro -C main include bullseye-wikimedia WMF_BUILD_DIR/packagename_version_amd64.changes

2. Log your uploads in #wikimedia-operations on IRC using !log

Upload to a component directory

Sometimes we build packages for specific purposes thus we don't want them to be available under the main component. To create a new component, you simply need to prepare a patch for puppet/modules/aptrepo/files/distributions-wikimedia. After the component is available on install* servers, you can upload to it by running: 

# reprepro -C component/thumbor include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes

Note: if you get an error like Error opening config file './conf/distributions': No such file or directory(2), then you forgot to do sudo -H bash

Create a Debian patch

Sometimes we might wish to add a fix to the software we are packaging, not included in the original source. Since we are using git to store our source code and track changes to it, we don't need to bother with quilt and source packages ourselves. Instead, we can simply make changes on our packaging git branch as if it were a normal git repository, as described in Debian_packaging_with_dgit_and_CI#Making_Changes.

Browse current package source

Our Debian packages are maintained in git repositories, so to review the code for a package, git clone the relevant repository from WMF gitlab. The branch naming convention is described elsewhere, but briefly, branches called dgit/bookworm track the relevant Debian release, and branches called wikimedia-bookworm are packaging branches.

For older packages that were maintained using a source-package-based workflow (rather than a git-based one), you need to instead use dget to retrieve the source:

Retrieving old-style source packages using dget

Getting source from Debian

Always use dgit clone to fetch sources from Debian.

Troubleshooting

Dependency issues with debhelper when backporting

Sometimes the source package you are backporting depends upon a newer version of debhelper than is available in the target distribution: 

 pbuilder-satisfydepends-dummy : Depends: debhelper (>= 9.20160709) but 9.20150101+deb8u2 is to be installed.

Adding BACKPORTS=yes to the beginning of your pdebuild invocation will often fix this.

See also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Debian_packaging&oldid=2203197"