Git-buildpackage

From Wikitech
This is a pretty thorough tutorial on git-buildpackage with a lot of info. If you just want to quickly have an environment to build a package cleanly and get going look at https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/ and apply the role::package::builder on a labs VM

Using git-buildpackage with source that already has a debian/ directory

(extracted from #Experiments below.)

Initially cloning the git repository from Upstream

  • Clone the repository from upstream if it doesn't already exist in Gerrit, e.g. from git.debian.org:
$ cd ~/git/debs/
$ git clone git://git.debian.org/git/pkg-puppet/repo.git
  • Set git username/email
$ git config --global user.name "Your Name"
$ git config --global user.email you@example.com
  • Set up the local branches upstream and pristine-tar from the remote branches (but don't switch to them):
$ git branch upstream origin/upstream
$ git branch pristine-tar origin/pristine-tar
  • Build a source package:
$ gbp buildpackage --git-export-dir=/var/tmp/build-area/packagename -S
  • Build the binary packages with Pbuilder:
# pbuilder build --basetgz /var/cache/pbuilder/lucid.tgz /var/tmp/build-area/packagename/packagename_version.dsc

Note: Applying role::package::builder on a labs VM will pre-populate a pbuilder/cowbuilder environment for you. Documentation on it is kept up to date at: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/package_builder/

Making changes and testing snapshot builds

  • Make changes to the package files
  • Make commits or amended commits (uncommitted changes are not exported to the build directory)
  • Build snapshot builds for testing with:
$ git-dch --snapshot
$ gbp buildpackage --git-export-dir=/var/tmp/build-area/packagename -S
# pbuilder --build --basetgz /var/cache/pbuilder/lucid.tgz /var/tmp/build-area/packagename/packagename_version.dsc

Doing a release

When a package is ready to be released, use the following steps: 

$ git-dch -R -N version
  • Commit the debian/changelog file (or amend a previous commit)
  • Build the final source package with --git-tag:
$ gbp buildpackage --git-tag --git-export-dir=/var/tmp/build-area/packagename -S
  • Build the binaries with Pbuilder as usual

Pushing changes into Gerrit

Push changes for each of the three branches (if applicable): 

$ git push gerrit master:refs/for/master
$ git push gerrit upstream:refs/for/upstream
$ git push gerrit pristine-tar:refs/for/pristine-tar

Pushing the repository into Gerrit

You should push any repository with local changes into Gerrit.

  • Create a new repository, operations/debs/packagename using the following Gerrit command:
$ ssh username@gerrit.wikimedia.org -p 29418 'gerrit create-project -d "Package packagename" operations/debs/packagename -o ldap/ops -p operations/debs'
  • Create a new remote called gerrit:
$ git remote add gerrit ssh://username@gerrit.wikimedia.org:29418/operations/debs/packagename
  • Push the three local branches (master, upstream and pristine-tar) into the fresh remote repository:
$ git push gerrit --all

How to build a Python deb package using git-buildpackage

First, it might be good to familiarize yourself with git-buildpackage. This manual is actually very helpful: http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html

Particularly, read the bit about different ways to import packages. The package that you are importing may vary in its own packaging. Do they release tarballs? Are you importing from an upstream git repository? Etc. etc.

Building a python package from a release tarball

I just built python-jsonschema. I'll use it as an example case.

Prep your git repository

Download the release tarball: 

 wget https://pypi.python.org/packages/source/j/jsonschema/jsonschema-1.1.0.tar.gz

If you don't yet have a repository in gerrit, you'll need to create one: 

 ssh username@gerrit.wikimedia.org -p 29418 'gerrit create-project -d "Package python-jsonschema" operations/debs/python-jsonschema -o ldap/ops -p operations/debs'
 git remote add gerrit ssh://username@gerrit.wikimedia.org:29418/operations/debs/python-jsonschema

The gerrit repository we just created is empty. Let's start a new one locally. 

 mkdir -p python-jsonschema
 cd python-jsonschema
 git init .

We are importing from a tarball, and not a git repository and git-import-orig wants an empty 'upstream' branch to exist on our repository. See: http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.import.html#AEN212 

 git checkout --orphan upstream
 git commit --allow-empty -m 'Initial upstream branch.'
 # switch back to master
 git checkout -f master


NOTE: master will be the name of the git-buildpackage specific 'debian-branch'. IF you want to use a different branch for this name, you will have to tell git-buildpackage via a config file, or via the CLI options. git-buildpackage suggests using master, so we'll stick with it.


Import the tarball

Since we're building a python module, be sure to name the package with the 'python-' prefix. We're also using --pristine-tar as git-buildpackage recommends. 

 git-import-orig  --pristine-tar ../jsonschema-1.1.0.tar.gz
 What will be the source package name? [jsonschema] python-jsonschema
 What is the upstream version? [1.1.0]
 gbp:info: Importing '../jsonschema-1.1.0.tar.gz' to branch 'upstream'...
 gbp:info: Source package is python-jsonschema
 gbp:info: Upstream version is 1.1.0
 gbp:info: Merging to 'master'
 Merge made by the 'recursive' strategy.
 gbp:info: Successfully imported version 1.1.0 of ../jsonschema-1.1.0.tar.gz

Use stdeb to create the debian/ directory

Faidon says not to use stdeb, but it creates a decent debian/ directory that gives a good starting point. You can create your debian/ directory however you like. See python-jsonschema/debian for a decent version of a python package debian/ directory that will use dh_python2.

For our example, we'll use stdeb and then edit the debian/ files and use dh_python2 to build the package.

See: https://pypi.python.org/pypi/stdeb 

 python setup.py --command-packages=stdeb.command debianize

Since we don't like stdeb's output, follow these instructions to modify debian/ files to work with dh_python2:

http://wiki.debian.org/Python/TransitionToDHPython2

In summary:

  • changelog:
    • Change maintainer to you.
    • Change changelog message to 'Initial release.'
  • compat:
    • change to 10 (9 if you need to support jessie)
  • control:
    • Change maintainer to you.
    • debhelper (>= 10) (again, 9 for jessie)
    • Put relevant dependencies (from setup.py / requirements.txt) in Depends
      • If you are building for python3 as well, put those dependencies also in build depends.
    • Change Standards-Version to 3.9.5
    • Modify description to remove any special formatting (markdown, etc.)
  • rules
    • remove comments about stdeb
    • Change to use pybuild, e.g.:
#!/usr/bin/make -f

# The deb package will be named python3?-$PYBUILD_NAME
export PYBUILD_NAME=google-api
## Uncomment these if you need to disable tests during deb build.
## Disable tests so we don't need test dependencies at build time.
#export PYBUILD_DISABLE_python2.7=test
#export PYBUILD_DISABLE_python3=test

%:
	dh $@ --with python2,python3 --buildsystem=pybuild

( --with whatever versions of python you are building for.)

Add the debian/ directory to git: 

 git add debian
 git commit -m 'Initial deb packaging'

Build the package

Now try to build the package, yay! 

 mkdir -p ../build-area/python-jsonschema
 gbp buildpackage --git-export-dir=../build-area/python-jsonschema

Check to see that the .deb looks like it should: 

 dpkg -c ../build-area/python-jsonschema/*.deb

If that looks good, push everything to gerrit, minus your most recent commit (that one needs review).

Push to gerrit for review

Directly push the upstream and pristine-tar branches, those don't need review: 

 git push gerrit upstream
 git push gerrit pristine-tar

Directly push everything in master except for the commit in which you added the debian/ directory: 

 git push gerrit HEAD~1:refs/heads/master

Finally git-review (or whatever) to push the last unpushed commit for review: 

 git review

Add the reviewed package to apt.wikimedia.org

Once your review has been approved and merged, copy the files in build-area/python-jsonschema to install1002, and add the .changes file with reprepro. We need to add the --ignore=wrongdistribution flag, since we didn't build the package with a special wikimedia distribution name. 

 reprepro --ignore=wrongdistribution -C main include precise-wikimedia jsonschema_1.1.0-1_amd64.changes

Building a python package from an existing git repository

Based on [1] 

$ git clone https://github.com/sripathikrishnan/redis-rdb-tools
$ cd redis-rdb-tools
$ vim .git/config
 # hack out the branch "master" upstream tracking

# upstream doesn't have tags, hack these in
$ git reset --hard d2792069291484af1d9d64abd849aaa44ddf6211
$ git tag upstream/0.1.6

then follow the instructions above from 'Use stdeb to create the debian/ directory'.

Ottomata TODO: understand and write this section.

FAQ

dpkg-source complaining about local changes

You might have dpkg-source complains about local change. You can get them ignore by editing debian/source/options and add the file to ignore list: 

$ cat debian/source/options
extend-diff-ignore = '^\.gitreview$'

Reference: http://www.debian.org/doc/manuals/maint-guide/dother.en.html#sourceopt

Investigating build failures

If you're invoking git-buildpackage with --git-builder=pbuilder, the chroot where the package is built will be removed at the end of the process, leaving you without the chance of investigating the build failure: 

 -> Cleaning COW directory
  forking: rm -rf /var/cache/pbuilder/build//cow.25824

The following pbuilder hook can be installed under /var/cache/pbuilder/hooks/$DIST/ to run a shell in the chroot after a build failure: 

 #!/bin/bash
 # example file to be used with --hookdir
 #
 # invoke shell if build fails.
 apt-get install -y "${APTGETOPT[@]}" vim less 
 cd /tmp/buildd/*/debian/..
 /bin/bash < /dev/tty > /dev/tty 2> /dev/tty

Experiments

experiment build mathoid debian package on vagrant and publish to ppa

The result https://launchpad.net/~physikerwelt/+archive/ubuntu/mathoid/+packages 

 sudo apt-get install git-buildpackage
 sudo apt-get install libdistro-info-perl

without the libdistro-info-perl the changes between the last release could not be calculated 

 cd

it does not work in a folder that is mounted to the windows host machine (i.e. /vagrant/srv/mathoid can not be used) 

 mkdir container
 ls
 cd container/
 git clone https://github.com/physikerwelt/mathoid-server
 cd mathoid-server/
 npm install
 git-dch -R -N 0.2.8
 git commit -a -m "Update debian release notes"
 sudo apt-get install dh-make
 sudo apt-get install gnupg2
 gpg2 --gen-key
 apt-get install haveged

this command was executed in a second terminal window because it was hard to generate entropy 

 gpg2 --send-keys --keyserver=keyserver.ubuntu.com 7271B791
 vi test.msg

you get an encrypted email that contains a link that you have to click... you need to find some way to read the contents... probably there is also a plugin for the gmail webui... but I could not find it 

 gpg -d test.msg
 rm test.msg
 gbp buildpackage --git-tag -S -k7271B791

normally the k argument should not be required 

 sudo apt-get install dput
 dput ppa:physikerwelt/mathoid ../mathoid_0.2.8_source.changes
 git push origin debian/0.2.8

External links

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Git-buildpackage&oldid=2123305"