Debian packaging/Package your software as deb

When software we develop at Wikimedia needs to be packaged for Debian, you typically want minimal disruption to your development process. Many upstream Debian packages use a per-distribution branch to build the software for different Debian versions, but this is inconvenient if the only thing that has to change is the debian/changelog entry. The process described in this page lets you publish a Debian package from source code that you stored in a GitLab repository, without the need of any additional Git branches.

If you're repackaging a package from debian, adding patches or tweaking the packaging, you should follow the process described in Debian packaging with dgit and CI instead.

GitLab CI pipeline

When developing software, you will typically have a project-specific CI pipeline in GitLab, and any additional pipeline will have to be merged into it. To this end, we've developed a shared pipeline configuration that you can include in your existing GitLab CI pipeline. It will build a Debian package every time that both of the following conditions are true:

a commit is pushed or merged onto your main branch
there is a change to debian/changelog

The deb package will be built for the distribution indicated in the latest entry in the debian/changelog file (which must be in the format $distro-wikimedia, or the CI pipeline will fail).

Method 1: One commit for each distro release

There are 2 methods for building .debs - this one requires multiple commits in the debian/changelog file.

Example

Typically, you have your own CI tasks for your code already. So, just modify your .gitlab-ci.yml file to include the following:

# Based on https://wikitech.wikimedia.org/wiki/Debian_packaging/Package_your_software_as_deb
# you can have more stages, but these are mandatory or CI will fail
stages:
  - prepare
  - build
  - upload
  - release
  
# Again this just needs to be added to your own set of variables if you have them.
# Please note: this needs to be declared before the include.
variables:
  # Use backports when building debs
  USEBACKPORTS: "1"
  # Do not change this.
  WMF_CI_RELEASE_DEB: "0"

include:
  - project: 'repos/sre/wmf-debci'
    ref: main
    file:
      - 'wmfdeb.yml'
      
# Run the 'build_ci_deb' job on a Trusted Runner
# https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Trusted_Runners
build_ci_deb:
  tags:
    - trusted

To use Trusted Runners, you need to add your project to the gitlab-trusted-runner repo. Until you do that, you can comment out the "trusted" tag and the pipeline will run on a non-trusted runner. The package will still be created, but you won't be able to publish it to the APT repositories.

Local testing

You can use gitlab-ci-local to test the pipeline locally. You'll have to specify the Debian version you are targeting, for example:

$ gitlab-ci-local build_ci_deb --variable SUITE=trixie

If the pipeline completes successfully, you will find the built package under WMF_BUILD_DIR/.

How to publish the package to the APT repositories

There is a cronjob on the APT Staging repository that scans GitLab CI to fetch and import the new debs from the CI Artifacts to the APT Staging repository automatically. Given you want to import your packages into the main APT repository eventually, you need to also do the setup to promote them to the main repository.

Prerequisites

To publish the packages to the APT repository, the following two conditions need to be satisfied:

Trusted Runners are enabled for your repo in projects.json
The package name is included in the aptrepo configuration in Puppet

A typical workflow

Let's say we want to make a new release for vopsbot, and we want to package the software for both Debian Bullseye and Debian Bookworm.

Bookworm

First, we'll add an entry to debian/changelog for Bookworm using dch (if you're not using Debian on your laptop, you can run it using Docker):

dch -v 0.3.8-4 --distribution 'bookworm-wikimedia' 'Build 0.3.8-4 for bookworm'

# This will add this section to debian/changelog:
# vopsbot (0.3.8-4) bookworm-wikimedia; urgency=medium
# [...]

Push the change to main, and it will trigger the GitLab Pipeline that builds the bookworm package:
```
git add debian/changelog
git commit -m 'Build 0.3.8-4 for bookworm'
git push
```

Bullseye

Now you add another entry for Bullseye, using e.g. version ~deb11u1 (see this note on choosing version numbers), to debian/changelog:

dch -v 0.3.8-4~deb11u1 --distribution 'bullseye-wikimedia' --force-distribution --force-bad-version 'Rebuild for bullseye'

# This will add this section to debian/changelog:
# vopsbot (0.3.8-4~deb11u1) bullseye-wikimedia; urgency=medium
# [...]

Push the change to main, and it will trigger the GitLab Pipeline that builds the bullseye package:
```
git add debian/changelog
git commit -m 'Build 0.3.8-4~deb11u1 for bullseye'
git push
```

The important detail is that you need to perform separate push operations for each of the distributions you want to build for, as the pipeline only looks for the latest changelog entry.

Semi-automated upload to apt.wikimedia.org

Once your packages have been built by the GitLab Pipeline, you should check they've been imported from GitLab into the staging apt repository (this should happen automatically).

Then log into the server running the main apt repository (apt1002.wikimedia.org or apt2002.wikimedia.org) and run

$ sudo -i reprepro  --noskipold --restrict <package_name> checkupdate bookworm-wikimedia
# Example output:
Updates needed for 'bookworm-wikimedia|main|amd64':
'vopsbot': '0.3.8-1' will be upgraded to '0.3.9-1+deb12u1' (from 'vopsbot'):
 files needed: pool/main/v/vopsbot/vopsbot_0.3.9-1+deb12u1_amd64.deb

$ sudo -i reprepro --noskipold --restrict <package_name> update bookworm-wikimedia
Calculating packages to get...
Getting packages...
Installing (and possibly deleting) packages...
Exporting indices...

For details see APT_repository.

Method 2: One commit for all Debian versions

An alternative approach, if you have to build for multiple distributions, is to build them all at once from a single commit in the main branch.

dch is used during the CI run to manipulate the changelog as needed.

Example: conftool

stages:
  #- test     # uncomment to run unit tests
  - prepare
  - build
  - upload
  - release

# Just a test example, add your unit test here
# test:
#   stage: test
#   image:
#     name: docker-registry.wikimedia.org/bookworm
#   script:
#     - DEBIAN_FRONTEND=noninteractive apt-get update -q -y && apt-get install -q -y tox
#     - tox


# # Deb pkg build - do not change anything below # #

variables:
  WMF_CI_RELEASE_DEB: "0"
  # Use backports when building debs
  USEBACKPORTS: "1"

include:
  - project: 'repos/sre/wmf-debci'
    ref: main
    file:
      - 'wmfdeb.yml'

# This builds Trixie (13). The changelog file must target `trixie-wikimedia`
build_ci_deb:
  tags:
    - trusted

# Backports (see below)
.backport:
  extends: build_ci_deb
  stage: build
  tags:
    - trusted
  before_script:
    - echo "[dch] Setting version to $(dpkg-parsechangelog -S version)~deb${DEB_VERSION_NUMBER}u1"
    - echo "[dch] Setting distribution to ${DISTRO_NAME}"
    - dch -v "$(dpkg-parsechangelog -S version)~deb${DEB_VERSION_NUMBER}u1"
      -D "${DISTRO_NAME}"
      "Backport to ${DISTRO_NAME}"

# Bullseye (11)
build_ci_deb_bullseye:
 extends: .backport
 image: docker-registry.wikimedia.org/wmf-debci-bullseye
 variables:
   SUITE: "bullseye"
   DISTRO_NAME: "bullseye-wikimedia"
   DEB_VERSION_NUMBER: "11"

# Bookworm (12)
build_ci_deb_bookworm:
 extends: .backport
 image: docker-registry.wikimedia.org/wmf-debci-bookworm
 variables:
   SUITE: "bookworm"
   DISTRO_NAME: "bookworm-wikimedia"
   DEB_VERSION_NUMBER: "12"

Your debian/changelog would then need only a plain version number e.g.

conftool (6.1.0) trixie-wikimedia; urgency=medium

The build_ci_deb block builds the Debian version in the debian/changelog file (Trixie) - make sure you have build_ci_deb_DISTRONAME blocks for the *other* Debian versions

Caution: SUITE DISTRO_NAME DEB_VERSION_NUMBER do magic stuff, do not rename them

In case of issues look at the [dch] log lines on gitlab

Upload to apt.wikimedia.org

Log on the staging hosts and review:

ssh apt-staging2001.codfw.wmnet 

grep -i <PKGNAME> /var/log/gitlab-package-puller/syslog.log

curl https://apt-staging.wikimedia.org/wikimedia-staging/dists/trixie-wikimedia/main/binary-amd64/Packages.gz -s | gunzip | grep 'Package: <PKGNAME>' -A20

ls -ltr /srv/aptrepo/wikimedia-staging/pool/main/<TAB_COMPLETE_HERE>

You should be seeing packages imported for each Debian version.

ssh apt1002.wikimedia.org

$ sudo -i reprepro  --noskipold --restrict <PKGNAME> checkupdate bookworm-wikimedia
# Example output:
Updates needed for 'bookworm-wikimedia|main|amd64':
'vopsbot': '0.3.8-1' will be upgraded to '0.3.9-1+deb12u1' (from 'vopsbot'):
 files needed: pool/main/v/vopsbot/vopsbot_0.3.9-1+deb12u1_amd64.deb

$ sudo -i reprepro --noskipold --restrict <PKGNAME> update bookworm-wikimedia
Calculating packages to get...
Getting packages...
Installing (and possibly deleting) packages...
Exporting indices...