Gerrit

We use Gerrit for our code review system for Git. It supersedes the CodeReview MediaWiki extension we have used for Subversion.

Quick links

• Gerrit
• Built-in Gerrit documentation
• Gerrit test install (hosted in a Cloud VPS project)
• Commit message guidelines
• Gerrit JavaMelody Monitoring
• Gerrit test instance

Details

Gerrit is installed on cobalt in the prefix /var/lib/gerrit2. A mostly-cold replica runs at gerrit2001.

It is a Java daemon which listens for HTTP connections (port 8080) and SSH connections (port 29418). Apache proxies the relevant URLs on port 80 and 443 through to Gerrit. The SSH port provides a restricted shell for Git checkouts and administrative commands.

It uses the m2 DB cluster, database name "reviewdb", for most storage. Git repositories are stored at `/srv/gerrit/git/`

It uses the LDAP instance shared with Wikimedia Cloud Services and various other things, for authentication. Accounts under ou=people,dc=wikimedia,dc=org and groups under ou=groups,dc=wikimedia,dc=org are exposed to Gerrit. The Gerrit account name is the "cn" field in LDAP. In order to log in to Gerrit, a user needs to already have a Wikimedia developer account. See Help:Getting Started for the process of getting a developer account.

To find out what version of Gerrit is running, you can use either of the following two commands:

java -jar /var/lib/gerrit2/review_site/bin/gerrit.war version
ssh -p 29418 gerrit.wikimedia.org gerrit version

It is 2.14.8-22-g07c8aa9910 at the time of writing.

Using git

See mw:Gerrit/Tutorial and mw:Gerrit/Advanced usage.

Database install

Database installation is handled by Gerrit installation process, you simply need a user named 'gerrit2' with appropriate privileges. Make sure to set the charset of the database to utf8.

Replace <PASSWORD> below and apply:

 CREATE USER 'gerrit2'@'localhost' IDENTIFIED BY <PASSWORD>;
 CREATE DATABASE reviewdb;
 ALTER DATABASE reviewdb charset=utf8;
 GRANT ALL ON reviewdb.* TO 'gerrit2'@'localhost';
 FLUSH PRIVILEGES;

Upgrade procedures

Prerequisites

1. Install OpenJDK 8 and git (openjdk-8-jdk and git packages on Debian)
2. Install Bazel as of Februrary 2019 this requires bazel 0.22.0

Update our repository

1. Clone gerrit and submodules and add upstream as a remote

   git clone https://gerrit.wikimedia.org/r/operations/software/gerrit
   cd gerrit
   git remote add upstream https://gerrit.googlesource.com/gerrit
   

2. Checkout the stable-X branch that tracks the current upstream release. As of 2018-08-01 X is 2.15

   git checkout stable-2.15 origin/stable-2.15
   

3. Fetch upstream and fast-forward our branch to the latest tag and push the stable-X branch directly to gerrit

   git fetch --tags upstream/2.15
   git merge --ff-only v2.15.8
   git push origin stable-2.15
   

4. Checkout wmf/stable-X and merge the latest tag, push the resulting SINGLE merge commit up for review

   git checkout wmf/stable-2.15
   git submodule update --init --recursive
   

   Now the world has frowned on your endeavor. You need to update submodules — oh dear! The easiest way I've found to do this is to make (gasp!) a shell script. Here's a file to put into your path called gerrit-submodule-update.sh

   #!/usr/bin/env bash
   
   VERSION="$1"
   BRANCH="$2"
   
   if [[ "$path" = "plugins/go-import" ]]; then
     return
   fi
   if [[ "$path" = "plugins/wikimedia" ]]; then
     return
   fi
   git fetch
   if git rev-parse "$VERSION" &>/dev/null; then
       git checkout "$VERSION"
   else
       git checkout "$BRANCH"
       git pull --ff-only
   fi
   

   Now updating submodules becomes kinda-sorta easier:

   # You are in a maze of special little cases all the same
   git submodule foreach 'gerrit-submodule-update.sh "v2.15.8" "stable-2.15"'
   # we maintain our own fork of go-import for the time being (as of 2019-01-11), so handle that seperately
   cd ./plugins/go-import
   git fetch upstream
   git merge --ff-only upstream/stable-2.15
   git push origin stable-2.15
   cd ../..
   # we "maintain" (which may be too strong of a word) our own plugin
   cd ./plugins/wikimedia
   git fetch
   git checkout master
   cd ../..
   git merge v2.15.8
   # Resolve any conflicts that may arise
   

5. Push for review. At this point, you will likely be ahead of wmf/stable-2.15 by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:

   git push origin HEAD:refs/for/wmf/stable-2.15/v2.15.8
   

Build Gerrit

1. Checkout the wmf/stable-X as of 2018-08-01 X is 2.15

   git checkout -b wmf/stable-2.15 origin/wmf/stable-2.15
   

2. Build the bazel-bin/release.war

   bazel build release
   

Troubleshooting

Current Bazel version is 0.18.1, expected at least 0.19.0 

Bazel updates all the time, update your bazel version from the bazel website (ensure you verify the version signature with the [ https://bazel.build/bazel-release.pub.gpg bazel-release.pub.gpg] key

curl: (35) error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol 

bazel shells out to curl to download things and, sadly, repo.eclipse.org only supports TLSv1 openssl s_client -connect repo.eclipse.org:443 -tls1. You can modify your openssl settings in /etc/ssl/openssl.cnf. If your MinProtocol is set to TLSv1.2 (because you're awesome) — change it to TLSv1 (because not everyone is awesome). Be sure to change it back when you're done.

Upload artifacts

1. Copy the resulting bazel-bin/release.war to somewhere safe
2. Checkout the deploy/wmf/stable-X branch that aligns with the build branch
3. Copy in the release.war you saved as ./gerrit.war
4. Unzip the war file to a temporary location
5. Copy all the files in WEB-INF/plugins/* over to ./plugins in the deploy branch
6. Commit the changes, upload to Gerrit, etc
7. Run ./upload_artifacts.py --version=2.15.2 gerrit.war plugins/foo.jar ... for the core application and all updated plugins (cf: Archiva, specifically setting up your ~/.m2/settings.xml

   Note: Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins so I wrote the tool above

Deploying

1. SSH to the deployment master
2. Navigate to /srv/deployment/gerrit/gerrit
3. Fetch & checkout the appropriate deploy/wmf/stable-X branch
4. Deploy to gerrit2001 to ensure that there are no errors with git-fat et al scap deploy -l gerrit2001.wikimedia.org 'Gerrit to [version] on gerrit2001'
5. Deploy to cobalt scap deploy -l cobalt.wikimedia.org 'Gerrit to [version] on cobalt'
6. If you're only deploying plugins, you're done, otherwise SSH to the Gerrit master and issue sudo service gerrit restart

Upgrading the database

When upgrading between major versions (or minor versions that have schema changes--for now), follow these steps:

1. Back up the database (reviewdb on m2) just in case

   Note: This is going away soon

2. Stop puppet
3. Stop the gerrit daemon
4. Start gerrit using the "init" flag. `cd /var/lib/gerrit2/review_site/ && sudo java -jar bin/gerrit.war init --batch --no-auto-start --skip-all-downloads`

   Note: Yes, this requires sudo. We do this because we don't want gerrit to actually be able to write to its config files, but the "init" process requires it. Puppet will fix permissions later.

5. Re-enable & then run puppet. This will fix permissions, the config file back to exactly how we expect, and then restart Gerrit for us

Creating new repositories

See mw:Git/Creating new repositories. Those without administrative access to Gerrit should use mw:Gerrit/New repositories instead.

Inspecting an account settings

Since Gerrit 2.15, accounts are stored in All-Users.git. One first need to find the account reference, that can be done by checking which references a user can read on All-Users.git:

$ ssh -p 29418 gerrit.wikimedia.org gerrit ls-user-refs --project All-Users --user 'johndoe@wikimedia.org' 
refs/meta/config
refs/users/34/1234
refs/users/self
$

Note: for accounts having extended permissions on All-Users.git, that will show all references which is not useful.

Then on the server side (or remotely if you have privileges), you can list the configuration files and dump their content:

$ cd /srv/gerrit/git/All-Users.git

$ git ls-tree refs/users/34/1234
100644 blob xxx account.config
100644 blob yyy authorized_keys
100644 blob zzz preferences.confg
100644 blob ddd watch.config

$ git show refs/users/34/1234:authorized_keys
ssh-rsa AAAAxyz... RSA-1024

The All-Users database can be browsed locally via git. The easiest way to find a user given the All-Users git repo is to git-grep for the username or email in the refs/meta/external-ids ref:

$ mkdir All-Users && cd All-Users && git init
$ git pull https://gerrit.wikimedia.org/r/All-Users refs/meta/external-ids
$ git grep johndoe

Disabling / Blocking an account

This action is limited to users in the ldap/ops or capability-access-database groups.

If an account needs to be disabled for some reason, there's three steps to complete:

• remove the ssh keys
• set the account to inactive
• make all Gerrit users re-login

Then flush the web_sessions caches to make everyone re-login (aka, the sledgehammer to log one user out):

$ ssh -p 29418 gerrit.wikimedia.org gerrit flush-caches --cache web_sessions

Instead of setting account inactive via gsql you can alternatively use this command:

$ssh gerrit.wikimedia.org -p 29418 gerrit set-account --inactive <Username here>

Tasks management

Gerrit queue tasks, given you have the appropriate permission you can run either of:

$ ssh -p 29418 gerrit.wikimedia.org ps -w
$ ssh -p 29418 gerrit.wikimedia.org gerrit show-queue -w

Tasks can be stuck which starve Gerrit processing. For example people would no more be able to fetch. In such a case, granted you are in the Administrators group you can kill a task by id:

$ ssh -p 29418 gerrit.wikimedia.org kill <SOMEID>

Replication

Forcing Replication re-runs

When forcing a replication run through gerrit's replication plugin, stay clear of the --all switch, as it breaks renamed projects on github.

For example, both gerrit and github have a mediawiki repository. But github's mediawiki repository is actually mediawiki/core in gerrit. As gerrit also has a mediawiki repository, forcing replication on all projects would push gerrit's mediawiki repository onto github's mediawiki repository. Thereby, github's mediawiki repository falls over, and for examples the release tags from github's mediawiki are gone (cf. task T100409).

This currently (2015-05-28) affects us for at least for the following repos:

Gerrit's name	Overloaded name on GitHub
mediawiki/core	mediawiki
VisualEditor/VisualEditor	VisualEditor
oojs/core	oojs

If you by accident forced replication with --all, force replication of the effective gerrit repos to make up for it (i.e.: mediawiki/core, VisualEditor/VisualEditor, ...)

Github

As of July 2018 replication to github requires a repository to be manually created on github using the naming scheme gerrit expects, i.e. / replaced with - (FrEx: operations/debs/trafficserver -> operations-debs-trafficserver.

Github owners will be able to create these repositories.

Once a repository is created, replication to that repo can be forced using the gerrit replication ssh command.

$ ssh -p 29418 gerrit.wikimedia.org replication start operations/debs/trafficserver --wait

ORES

ORES mirroring setup is dangerous and unusual, but should, nevertheless, be documented.

The canonical repository for ORES lives on github. This is repository is observed by diffusion. This can done in the Manage Repository section of diffusion code browser. From there, phabricator "Mirrors" that repository (also available in diffusion management settings) to gerrit.

This setup is discouraged.

How to setup Gerrit in a Cloud VPS project

See Gerrit/Running in Cloud VPS.

Monitoring

JavaMelody

As of November 2018 gerrit has the JavaMelody plugin installed. This can be used to monitor the state of threads in the JVM running gerrit.

It is currently accessible to gerrit admins at: https://gerrit.wikimedia.org/r/monitoring

Java trace

   sudo -u gerrit2 /usr/lib/jvm/java-8-openjdk-amd64/bin/jstat -gcutil `pidof GerritCodeReview` 1000 3

Logstash

Gerrit's logs are sent to logstash using log4j. You can find its logs by searching with type:log4j.

Logs are also available on the gerrit servers at: /var/lib/gerrit2/review_site/logs/

See also

• gerrit-wm
• Gerrit Notification Bot