Jump to content

Debian packaging/Package your software as deb

From Wikitech

When software we develop at Wikimedia needs to be packaged for Debian, you typically want minimal disruption to your development process. Many upstream Debian packages use a per-distribution branch to build the software for different Debian versions, but this is inconvenient if the only thing that has to change is the debian/changelog entry. The process described in this page lets you publish a Debian package from source code that you stored in a GitLab repository, without the need of any additional Git branches.

If you're repackaging a package from debian, adding patches or tweaking the packaging, you should follow the process described in Debian packaging with dgit and CI instead.

GitLab CI pipeline

When developing software, you will typically have a project-specific CI pipeline in GitLab, and any additional pipeline will have to be merged into it. To this end, we've developed a shared pipeline configuration that you can include in your existing GitLab CI pipeline. It will build a Debian package every time that both of the following conditions are true:

  • a commit is pushed or merged onto your main branch
  • there is a change to debian/changelog

The deb package will be built for the distribution indicated in the latest entry in the debian/changelog file (which must be in the format $distro-wikimedia, or the CI pipeline will fail).

Example

Typically, you have your own CI tasks for your code already. So, just modify your .gitlab-ci.yml file to include the following: 

# Based on https://wikitech.wikimedia.org/wiki/Debian_packaging/Package_your_software_as_deb
# you can have more stages, but these are mandatory or CI will fail
stages:
  - prepare
  - build
  - upload
  - release
  
# Again this just needs to be added to your own set of variables if you have them.
# Please note: this needs to be declared before the include.
variables:
  # Use backports when building debs
  USEBACKPORTS: "1"
  # Do not change this.
  WMF_CI_RELEASE_DEB: "0"

include:
  - project: 'repos/sre/wmf-debci'
    ref: main
    file:
      - 'wmfdeb.yml'
      
# Run the 'build_ci_deb' job on a Trusted Runner
# https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Trusted_Runners
build_ci_deb:
  tags:
    - trusted
To use Trusted Runners, you need to add your project to the gitlab-trusted-runner repo. Until you do that, you can comment out the "trusted" tag and the pipeline will run on a non-trusted runner. The package will still be created, but you won't be able to publish it to the APT repositories.

Local testing

You can use gitlab-ci-local to test the pipeline locally. You'll have to specify the Debian version you are targeting, for example: 

$ gitlab-ci-local build_ci_deb --variable SUITE=trixie

If the pipeline completes successfully, you will find the built package under WMF_BUILD_DIR/.

How to publish the package to the APT repositories

There is a cronjob on the APT Staging repository that scans GitLab CI to fetch and import the new debs from the CI Artifacts to the APT Staging repository automatically. Given you want to import your packages into the main APT repository eventually, you need to also do the setup to promote them to the main repository.

Prerequisites

To publish the packages to the APT repository, the following two conditions need to be satisfied:

A typical workflow

Let's see how a typical workflow will pan out:

Let's say we want to make a new release for vopsbot, and we want to package the software for both Debian Bullseye and Debian Bookworm.

Bookworm

  • First, we'll add an entry to debian/changelog for Bookworm, eg.
 vopsbot (0.3.8-4) bookworm-wikimedia; urgency=medium
 [...]
  • push the change to main
  • The bookworm package will be built:
dch -v 0.3.8-4
git add debian/changelog
git commit -m 'Build 0.3.8-4 for bookworm'
git push
# This triggers a pipeline like https://gitlab.wikimedia.org/repos/sre/vopsbot/-/pipelines/37699

Bullseye

 vopsbot (0.3.8-4~deb11u1) bullseye-wikimedia; urgency=medium
 [...]
  • push the change to the main branch of the repository.
  • The bullseye package will be built:
# Adds a new changelog entry with your message of choice
DEBEMAIL=<changeme>@wikimedia.org dch -v 1.2.3~wmf1 'Rebuild for bullseye' \
 --distribution 'bullseye-wikimedia' --force-distribution
DEBEMAIL=<changeme>@wikimedia.org dch --release
git diff
git commit -m 'Build 1.2.3~wmf1 for bullseye'
git push

# This triggers a pipeline like https://gitlab.wikimedia.org/repos/sre/vopsbot/-/pipelines/37702

The important detail is that you need to perform separate push operations for each of the distributions you want to build for, as the pipeline only looks for the latest changelog entry.

Semi-automated upload to apt.wikimedia.org

Once your packages have been built, you should check they've been uploaded to the staging apt repository, then log into the server running the main apt repository (apt1002.wikimedia.org or apt2002.wikimedia.org) and run 

$ sudo -i reprepro  --noskipold --restrict <package_name> checkupdate bookworm-wikimedia
# Example output:
Updates needed for 'bookworm-wikimedia|main|amd64':
'vopsbot': '0.3.8-1' will be upgraded to '0.3.9-1+deb12u1' (from 'vopsbot'):
 files needed: pool/main/v/vopsbot/vopsbot_0.3.9-1+deb12u1_amd64.deb

$ sudo -i reprepro --noskipold --restrict <package_name> update bookworm-wikimedia
Calculating packages to get...
Getting packages...
Installing (and possibly deleting) packages...
Exporting indices...

For details see APT_repository

Building for multiple distributions at once

An alternative approach, if you have to build for multiple distributions, is to build them all at once by doing tricks with the changelog file in CI, see for example how conftool does it.

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Debian_packaging/Package_your_software_as_deb&oldid=2355832"