Debian Packaging

From Wikitech
Jump to navigation Jump to search

These are runbooks related to Debian packaging for Wikimedia Foundation production.

Rebuilding a package

Process to repackage a Debian package and push it to wmf apt repo.

Example

To repackage python-libthumbor:

  1. Login to build2001.codfw.wmnet
  2. Run the following to download python-libthumbor's source package:
    $ apt-get source libthumbor=1.3.2-0+wmf1
    We should now have 3 files and 1 directory in our working directory:
    libthumbor_1.3.2.orig.tar.gz
    Source code of the upstream authors
    libthumbor_1.3.2-0+wmf1.dsc
    Meta file describing which files are around
    libthumbor_1.3.2-0+wmf1.debian.tar.xz
    Debian-specific changes
    libthumbor-1.3.2
    Extracted source package
  3. Note the Debian-specific files in the libthumbor-1.3.2/debian directory. For more information about these files, consult Debian's manual detailing Required files under the debian directory and Other files under the debian directory.
  4. Navigate to the libthumbor-1.3.2 directory and use dch to add a new changelog entry to the top of the debian/changelog: 
    $ cd libthumbor-1.3.2
$ dch -i
    • Add the new version number, making sure that it's higher than the previous version.
    • Change UNRELEASED to VERSION_CODENAME-wikimedia where VERSION_CODENAME is the targeted Debian version.
    • Fill in a clear and accurate list of what changes have occurred since the last entry.
    An example entry might look like:
    libthumbor (1.3.2-0+wmf1+stretch1) stretch-wikimedia; urgency=low
 
 * Rebuild for stretch
 
-- Effie Mouzeli <effie@wikimedia.org>  Tue, 20 Nov 2018 13:42:24 +0000
 
libthumbor (1.3.2-0+wmf1) unstable; urgency=low
 
 * Initial release. (Closes: #783577)
 
-- Gilles Dubuc <gilles@wikimedia.org>  Thu, 19 May 2016 15:30:16 +0000
    When we rebuild for a new distro we use ~, not + like in the example in that page, the difference being 1.5-1~something is less than 1.5-1 so if 1.5-1 ever appears on official buster, it will supersede our package.
  5. Submit a patch to Gerrit and wait for approval before merging.
  6. Build:
    • To use packages from the VERSION_CODENAME repo run DIST=VERSION_CODENAME pdebuild
    • To include packages from the VERSION_CODENAME-wikimedia repo, run DIST=VERSION_CODENAME-wikimedia pdebuild
    • If building for a new Debian version do something like DIST=VERSION_CODENAME WIKIMEDIA=yes pdebuild --debbuildopts -sa
  7. Find the build results in /var/cache/pbuilder/result/VERSION_CODENAME-amd64/:
    • libthumbor_1.3.2-0+wmf1+VERSION_CODENAME1_amd64.buildinfo
    • libthumbor_1.3.2-0+wmf1+VERSION_CODENAME1_amd64.changes
    • libthumbor_1.3.2-0+wmf1+VERSION_CODENAME1.debian.tar.xz
    • libthumbor_1.3.2-0+wmf1+VERSION_CODENAME1.dsc
    • python-libthumbor_1.3.2-0+wmf1+VERSION_CODENAME1_all.deb
  8. Upload your artifacts to the Wikimedia repository.

Download source packages from snapshot.debian.org

If you are looking to build a different version than the one provided by a source package, you can have a look at Debian snapshots. Everything you need will be downloaded for you by pointing dget to the relevant .dsc file: 

$ dget https://snapshot.debian.org/archive/debian-debug/20180826T210249Z/pool/main/libr/librsvg/librsvg_2.40.20-3.dsc

Package-specific deviations

This section needs expansion; Why do we need to do these?
  • If we are building a new package where we are the owners, like python-thumbor-wikimedia, we should use native instead of quilt in debian/source/format
  • If we are to use packages from eg php72 component, this can be done by using PHP72=yes env var before building.

Test with piuparts

Can be useful sometime to test how the just-build package behaves in terms of installing/upgrading/reinstalling. For this task the piuparts can help us.

piuparts is currently not installed on any buildhost. Consider installing locally, eg. on a VM on your computer

Once we have built the package we can test with: 

# piuparts -d <distribution> <package>

Eg. 

# piuparts -d bookworm varnishkafka_1.1.0-3_amd64.deb

If we want to test all the packages built from a single source package (ex. because there are some dependencies between them) we can pass directly the .changes file: 

# piuparts -d bookworm varnishkafka_1.1.0-3_amd64.changes

piuparts will try to install/upgrade/purge all binary packages listed in the .changes file.

If we want to test a single package against multiple distributions we can use: 

# piuparts -d bullseye -d bookworm varnishkafka_1.1.0-3_amd64.changes

If we want to include an extra repository (eg. for getting dependencies needed by our newly-built package) we can use something like: 

# piuparts -d bookworm --extra-repo='deb [ trusted=yes ] http://apt.wikimedia.org/wikimedia bookworm-wikimedia main' varnishkafka_1.1.0-3_amd64.deb

Upload to Wikimedia Repo

1. Login to apt1001.wikimedia.org, create a directory and copy all files from build2001 

$ ssh apt1001.wikimedia.org
$ mkdir libthumbor; cd libthumbor
$ rsync rsync://build2001.codfw.wmnet/pbuilder-result/stretch-amd64/*thum* .

2. Import! 

# reprepro -C main include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes

Note: We split the archives sometimes, i.e. we add packages to components instead of main

3. Log your uploads in #wikimedia-operations on IRC using !log

Upload to a component directory

Sometimes we build packages for specific purposes thus we don't want them to be available under the main component. To create a new component, you simply need to prepare a patch for puppet/modules/aptrepo/files/distributions-wikimedia. After the component is available on install* servers, you can upload to it by running: 

# reprepro -C component/thumbor include stretch-wikimedia libthumbor_1.3.2-0+stretch+wmf1_amd64.changes

Note: if you get an error like Error opening config file './conf/distributions': No such file or directory(2), then you forgot to do sudo -H bash

Create a Debian patch

Sometimes we might wish to add a fix to the software we are packaging, not included in the original source.

  1. Check if your source package is using quilt, Debian's implementation of patch series tracking: 
    $ cat debian/source/format
3.0 (quilt)
  2. Make the edits you wish to do for your fix and run the following command: 
    $ dpkg-source --commit  
librsvg-2.40.18/tests/crash.c
Enter the desired patch name: something-with-sense.patch
  3. Name your patch and use the spawned editor to make a description, for instance: 
    Description: add -exact in webp conversion
The -exact option was introduced in webp 0.5, which is only available on Stretch.

python-thumbor-wikimedia (2.2-1+deb9u1) stretch-wikimedia; urgency=low

  * Repackaging for stretch

Author: Effie Mouzeli <effie@wikimedia.org>
 ---
  4. Your patch will be saved under debian/patches/foo.patch. Build it in the usual way
    If you are uploading a package to a -wikimedia repository that is a patched version of the version originally in Debian proper, you should also pass --debbuildopts -sa to pdebuild. This will force it to include the .orig.tar.gz containing the source code as provided by the original developer.

Browse current package source

To review the active patches we have applied to a package, there are two approaches one can take: standalone using dget, or through apt source. The former has the benefit of not needing to modify any /etc/apt/ configuration in the environment where you do this. The latter has the benefit of using the same tooling and logic as installing a package would (and thus involve changing /etc/ files).

Standalone using dget

Through apt

  • Ensure /etc/apt/sources.list also includes the deb-src variant of the relevant sources.list lines. For example, when using component/php72 add the last line as well:
deb http://mirrors.wikimedia.org/debian/ buster main
deb http://security.debian.org/ buster/updates main
deb http://mirrors.wikimedia.org/debian/ buster-updates main
deb http://mirrors.wikimedia.org/debian/ buster-backports main contrib
deb http://apt.wikimedia.org/wikimedia buster-wikimedia main
deb http://apt.wikimedia.org/wikimedia buster-wikimedia component/php72

deb-src http://apt.wikimedia.org/wikimedia buster-wikimedia component/php72
  • Move to a temporary directory and run apt source <pgk-name>, e.g. apt source php7.2. This will use the same resolution logic as for apt-get install.
  • The package source is now in <pgk-name>-1.2.3-wmf1/, with a familiar "debian" subdirectory containing "debian/patches", etc.

See also

If a package is unchanged from upstream, then you can also browse the source online via Debian Salsa and its GitLab interface. For example:

https://salsa.debian.org/php-team/php/-/tree/debian/main/7.2/debian/patches

Troubleshooting

Dependency issues with debhelper when backporting

Sometimes the source package you are backporting depends upon a newer version of debhelper than is available in the target distribution: 

 pbuilder-satisfydepends-dummy : Depends: debhelper (>= 9.20160709) but 9.20150101+deb8u2 is to be installed.

Adding BACKPORTS=yes to the beginning of your pdebuild invocation will often fix this.

See also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Debian_Packaging&oldid=2097822"