This page is currently a draft.
More information and discussion about changes to this draft on the talk page.
This page contains information on the various packaging situations we do related to Toolforge. This refers specifically to Debian packages (.deb).
Approximate list of .deb packages we handle for Toolforge:
- tools-webservice: gerrit page: https://gerrit.wikimedia.org/r/admin/projects/operations/software/tools-webservice clone URL: https://gerrit.wikimedia.org/r/operations/software/tools-webservice
- jobutils/misctools: gerrit page: https://gerrit.wikimedia.org/r/admin/projects/labs/toollabs clone URL: https://gerrit.wikimedia.org/r/labs/toollabs
- TODO: any other?
sbuild tool is a simple and powerful tool to build debian packages.
Our packaging building hosts have sbuild ready to use. The first time you try to use sbuild you should add yourself to the sbuild unix group:
user@tools-package-builder-02:~$ sudo sbuild-adduser $LOGNAME [..] user@tools-package-builder-02:~$ newgrp sbuild
There should be environments to build packages for the most common distros: jessie, stretch, buster.
You can switch between them easily:
user@tools-package-builder-02:mypackage/$ sbuild -d jessie [..] user@tools-package-builder-02:mypackage/$ sbuild -d stretch [..] user@tools-package-builder-02:mypackage/$ sbuild -d buster [..]
Sometimes the build-deps required to clean the source tree are not installed outside the chroot. To workardound this, use the
--no-clean-source switch when calling sbuild.
Adding extra repos and packages is super easy, check the
--extra-repository option in the sbuild manpage.
Additional upstream documentation: https://wiki.debian.org/sbuild
TODO perhaps put here yuvi's method and bryan's method from Portal:Toolforge/Admin#Bd808's_method_(pdebuild).
This is a custom script to automate most of the build & publish steps of the process.
The steps are done in order:
- git stage: create a random directory in the package build server, git clone the source repository and checkout a given branch.
- sbuild stage: run sbuild to build the debian package.
- copy stage: copy the resulting .deb packages from the package build server into the aptly server at a random directory.
- aptly stage: add the .deb packages to a given repo and publish it.
- backup stage: run rsync from the aptly tree to NFS (optional).
- cleanup stage: delete random directories created by the script.
You can locate this script in the puppet tree: modules/toolforge/files/wmcs-package-build.py.
user@laptop:~$ modules/toolforge/files/wmcs-package-build.py --help usage: wmcs-package-build.py [-h] --git-repo GIT_REPO [--git-branch GIT_BRANCH] [--build-dist BUILD_DIST] [--build-host BUILD_HOST] [-a APTLY_DIST] [--aptly-host APTLY_HOST] [-b] [-d] Utility to build and upload a .deb package to aptly optional arguments: -h, --help show this help message and exit --git-repo GIT_REPO git repository URL with the source pkg. This script will do a fresh git clone of that repo. Typical value is something like: https://gerrit.wikimedia.org/r/operations/software/tools-webservice --git-branch GIT_BRANCH git branch to use to build the package from. Defaults to "master" --build-dist BUILD_DIST target distribution when building the package with sbuild. Defaults to "stretch" --build-host BUILD_HOST package build host. Typically a VM in CloudVPS with role::wmcs::toolforge::package_builder. Defaults to "tools-package- builder-02.tools.eqiad.wmflabs" -a APTLY_DIST, --aptly-dist APTLY_DIST target distribution in aptly. The resulting deb package will be uploaded to this distribution and then the repository will be published. Can be specified multiple times for multiple target distributions. If this argument is not provided, no aptly operations will be done. Example: -a stretch-tools -a stretch-toolsbeta --aptly-host APTLY_HOST aptly server host. Typically a VM in CloudVPS with role::wmcs::toolforge::services. Defaults to "tools-sge- services-03.tools.eqiad.wmflabs" -b, --no-backup If this option is present, this script won't backup aptly data over NFS -d, --dry-run Dry run: only show what this script would do, but don't do it for real
This script is supposed to run from your laptop, and it will start SSH connections to the different servers involved in building and publishing a deb package.
--help output should have enough information to know what every option does.
You are encouraged to use the
--dry-run first to know what the script would do (the actual SSH commands).
For the toolsbeta project
Try first in the toolsbeta project for testing purposes before releasing in the tools project.
For that, build the package and upload it to the -toolsbeta variants of the aptly repos:
user@laptop:~$ wmcs-package-build.py --git-repo https://gerrit.wikimedia.org/r/operations/software/tools-webservice --aptly-dist buster-toolsbeta --aptly-dist stretch-toolsbeta [..]
Then you can install the package in toolsbeta servers for testing.
For the tools project
Once testing is done and you are confident with a given package version, you can just run the script again and have the package upload to the final destination:
user@laptop:~$ wmcs-package-build.py --git-repo https://gerrit.wikimedia.org/r/operations/software/tools-webservice --aptly-dist jessie-tools --aptly-dist stretch-tools --aptly-dist buster-tools [..]
packaging good practices
Some good practices, advises and hints about our packaging workflows:
gbp dch --git-author --id-length=7 --since=...to generate debian/changelog file. This reads the git log to generate changelog entries. Make changes to d/changelog in a separate commit.
- Not on a Debian host? Try using Docker!
$ docker run -it --rm \ -v "$(pwd):/mnt" -v "$HOME/.gitconfig:/etc/skel/.gitconfig:ro" \ -e UID=$(id -u) -e GID=$(id -g) \ opxhub/gbp:buster \ bash -c 'gbp dch --git-author --id-length=7'
- Not on a Debian host? Try using Docker!
- when a new package is released (i.e, uploaded to a repository and/or distributed to clients) a git tag should be created in the git repository with the format debian/version.
- use the
sbuildtool to build the packages.