We use Gerrit for our code review system for Git. It supersedes the CodeReview MediaWiki extension we have used for Subversion.
- Built-in Gerrit documentation
- Gerrit test install (hosted in a Cloud VPS project)
- Commit message guidelines
- Gerrit JavaMelody Monitoring
- Gerrit test instance
Gerrit is installed on cobalt in the prefix /var/lib/gerrit2. A mostly-cold replica runs at gerrit2001.
It is a Java daemon which listens for HTTP connections (port 8080) and SSH connections (port 29418). Apache proxies the relevant URLs on port 80 and 443 through to Gerrit. The SSH port provides a restricted shell for Git checkouts and administrative commands.
It uses the m2 DB cluster, database name "reviewdb", for most storage. Git repositories are stored at `/srv/gerrit/git/`
It uses the LDAP instance shared with Wikimedia Cloud Services and various other things, for authentication. Accounts under ou=people,dc=wikimedia,dc=org and groups under ou=groups,dc=wikimedia,dc=org are exposed to Gerrit. The Gerrit account name is the "cn" field in LDAP. In order to log in to Gerrit, a user needs to already have a Wikimedia developer account. See Help:Getting Started for the process of getting a developer account.
To find out what version of Gerrit is running, you can use either of the following two commands:
java -jar /var/lib/gerrit2/review_site/bin/gerrit.war version ssh -p 29418 gerrit.wikimedia.org gerrit version
It is 2.14.8-22-g07c8aa9910 at the time of writing.
Database installation is handled by Gerrit installation process, you simply need a user named 'gerrit2' with appropriate privileges. Make sure to set the charset of the database to utf8.
Replace <PASSWORD> below and apply:
CREATE USER 'gerrit2'@'localhost' IDENTIFIED BY <PASSWORD>; CREATE DATABASE reviewdb; ALTER DATABASE reviewdb charset=utf8; GRANT ALL ON reviewdb.* TO 'gerrit2'@'localhost'; FLUSH PRIVILEGES;
- Install OpenJDK 8 and git (
gitpackages on Debian)
- Install Bazel as of Februrary 2019 this requires bazel 0.22.0
Update our repository
- Clone gerrit and submodules and add upstream as a remote
git clone https://gerrit.wikimedia.org/r/operations/software/gerrit cd gerrit git remote add upstream https://gerrit.googlesource.com/gerrit
- Checkout the
stable-Xbranch that tracks the current upstream release. As of 2018-08-01
git checkout stable-2.15 origin/stable-2.15
- Fetch upstream and fast-forward our branch to the latest tag and push the
stable-Xbranch directly to gerrit
git fetch --tags upstream/2.15 git merge --ff-only v2.15.8 git push origin stable-2.15
wmf/stable-Xand merge the latest tag, push the resulting SINGLE merge commit up for review
git checkout wmf/stable-2.15 git merge v2.15.12
- Resolve any conflicts that may arise
git submodule update --init --recursive
- Push for review. At this point, you will likely be ahead of
wmf/stable-2.15by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:
git push origin HEAD:refs/for/wmf/stable-2.15/v2.15.12
Now the world has frowned on your endeavor. You need to update submodules — oh dear! The easiest way I've found to do this is to make (gasp!) a shell script. Here's a file to put into your path called
#!/usr/bin/env bash VERSION="$1" BRANCH="$2" if [[ "$path" = "plugins/go-import" ]]; then return fi if [[ "$path" = "plugins/wikimedia" ]]; then return fi git fetch if git rev-parse "$VERSION" &>/dev/null; then git checkout "$VERSION" else git checkout "$BRANCH" git pull --ff-only fi
Now updating submodules becomes kinda-sorta easier:
# You are in a maze of special little cases all the same git submodule foreach 'gerrit-submodule-update.sh "v2.15.8" "stable-2.15"' # we maintain our own fork of go-import for the time being (as of 2019-01-11), so handle that seperately cd ./plugins/go-import git fetch upstream git merge --ff-only upstream/stable-2.15 git push origin stable-2.15 cd ../.. # we "maintain" (which may be too strong of a word) our own plugin cd ./plugins/wikimedia git fetch git checkout master cd ../.. git commit git push origin HEAD:refs/for/wmf/stable-2.15
- Checkout the
wmf/stable-Xas of 2018-08-01
git checkout -b wmf/stable-2.15 origin/wmf/stable-2.15
- Build the
bazel build release
Current Bazel version is 0.18.1, expected at least 0.19.0
- Bazel updates all the time, update your bazel version from the bazel website (ensure you verify the version signature with the [ https://bazel.build/bazel-release.pub.gpg bazel-release.pub.gpg] key
curl: (35) error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
- bazel shells out to curl to download things and, sadly,
repo.eclipse.orgonly supports TLSv1
openssl s_client -connect repo.eclipse.org:443 -tls1. You can modify your openssl settings in
/etc/ssl/openssl.cnf. If your
MinProtocolis set to
TLSv1.2(because you're awesome) — change it to
TLSv1(because not everyone is awesome). Be sure to change it back when you're done.
- Copy the resulting
bazel-bin/release.warto somewhere safe
- Checkout the
deploy/wmf/stable-Xbranch that aligns with the build branch
- Copy in the
release.waryou saved as
- Unzip the war file to a temporary location
- Copy all the files in
./pluginsin the deploy branch
- Commit the changes, upload to Gerrit, etc
./upload_artifacts.py --version=2.15.2 gerrit.war plugins/foo.jar ...for the core application and all updated plugins (cf: Archiva, specifically setting up your
- Note: Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins so I wrote the tool above
- SSH to the deployment master
- Navigate to
- Fetch & checkout the appropriate
- Deploy to gerrit2001 to ensure that there are no errors with git-fat et al
scap deploy -l gerrit2001.wikimedia.org 'Gerrit to [version] on gerrit2001'
- Deploy to cobalt
scap deploy -l cobalt.wikimedia.org 'Gerrit to [version] on cobalt'
- If you're only deploying plugins, you're done, otherwise SSH to the Gerrit master and issue
sudo service gerrit restart
Upgrading the database
When upgrading between major versions (or minor versions that have schema changes--for now), follow these steps:
- Back up the database (reviewdb on m2) just in case
- Note: This is going away soon
- Stop puppet
- Stop the gerrit daemon
- Start gerrit using the "init" flag. `cd /var/lib/gerrit2/review_site/ && sudo java -jar bin/gerrit.war init --batch --no-auto-start --skip-all-downloads`
- Note: Yes, this requires sudo. We do this because we don't want gerrit to actually be able to write to its config files, but the "init" process requires it. Puppet will fix permissions later.
- Re-enable & then run puppet. This will fix permissions, the config file back to exactly how we expect, and then restart Gerrit for us
Example migration ticket (cobalt -> gerrit1001): https://phabricator.wikimedia.org/T222391
Topic branch from gerrit1001 migration: https://gerrit.wikimedia.org/r/q/topic:%22gerrit1001%22+(status:open%20OR%20status:merged)
Follow these steps to migrate from one server to another:
Make sure you run rsync with --delete
- Rsync /srv/gerrit/git/ , /srv/gerrit/plugins and /var/lib/gerrit2/review_site/ from <old_host> to <new_host>
rsync --archive --verbose --delete /srv/gerrit/git/ rsync://gerrit1001.wikimedia.org/gerrit-data/git/
rsync --archive --verbose --delete /srv/gerrit/plugins/ rsync://gerrit1001.wikimedia.org/gerrit-data/plugins/
rsync --archive --verbose --delete /var/lib/gerrit2/review_site/ rsync://gerrit1001.wikimedia.org/gerrit-var-lib/
(Note: We are using rsync protocol directly and 'gerrit-data' and 'gerrit-var-'lib' are the names of rsync modules, not file system pathes. They are defined in "class profile::gerrit::migration".)
- Stop gerrit && disable puppet on <new_host>
- Create something similar to https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/535966/
- Stop puppet on <old_host> + <replica>
- Create something similar to https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/541110/
- Create something similar to https://gerrit.wikimedia.org/r/#/c/operations/dns/+/541111/
- Stop gerrit on <old_host>
- Repeat the rsync commands above.
- Rename /var/lib/gerrit2/review_site/data/javamelody/r_<old_host> to /var/lib/gerrit2/review_site/data/javamelody/r_<new_host> on <new_host>.
- Run puppet on <new_host> + <old_host>
- Start gerrit on <new_host>
- Hack DNS authdns-update to clone from gerrit-replica temporarily, deploy DNS change
- Manually copy apache2 site config for gerrit.wm.org with scp from <old_host> to <new_host>, restart apache
- Manually run command from list_mediawiki_extensions cron to create /var/www/mediawiki-extensions.txt
- Run the online reindexer
- Decom <old_host> (create a ticket like https://phabricator.wikimedia.org/T236187)
Creating new repositories
Inspecting an account settings
Since Gerrit 2.15, accounts are stored in
All-Users.git. One first need to find the account reference, that can be done by checking which references a user can read on All-Users.git:
$ ssh -p 29418 gerrit.wikimedia.org gerrit ls-user-refs --project All-Users --user 'firstname.lastname@example.org' refs/meta/config refs/users/34/1234 refs/users/self $
Note: for accounts having extended permissions on
All-Users.git, that will show all references which is not useful.
Then on the server side (or remotely if you have privileges), you can list the configuration files and dump their content:
$ cd /srv/gerrit/git/All-Users.git $ git ls-tree refs/users/34/1234 100644 blob xxx account.config 100644 blob yyy authorized_keys 100644 blob zzz preferences.confg 100644 blob ddd watch.config $ git show refs/users/34/1234:authorized_keys ssh-rsa AAAAxyz... RSA-1024
All-Users database can be browsed locally via git. The easiest way to find a user given the
All-Users git repo is to git-grep for the username or email in the
$ mkdir All-Users && cd All-Users && git init $ git pull https://gerrit.wikimedia.org/r/All-Users refs/meta/external-ids $ git grep johndoe
Become an Administrator
Being an administrator grants all global capabilities as described at Gerrit Access Control - global capabilities.
If right is given as a one off, one would most probably want to be removed from the group once the rights are no more required.
Disabling / Blocking an account
This action is limited to Gerrit administrators.
If an account needs to be disabled for some reason, there's three steps to complete:
- remove the ssh keys
- set the account to inactive
- make all Gerrit users re-login
Instead of setting account inactive via gsql you can alternatively use this command:
$ ssh gerrit.wikimedia.org -p 29418 gerrit set-account --inactive <Username here>
To enable the user again:
$ ssh gerrit.wikimedia.org -p 29418 gerrit set-account --active <Username here>
And flush the accounts cache on the replica:
$ ssh gerrit-replica.wikimedia.org -p 29418 gerrit flush-caches --cache accounts
Flush all sessions
Do not flush all sessions when banning a single user. As of May 2019 setting a user to inactive should invalidate their session automagically
web_sessions caches makes everyone re-login (aka, the sledgehammer). It should not be necessary in most circumstances; however, the command is still in this documentation in case it is needed in an unforeseen situation:
$ ssh -p 29418 gerrit.wikimedia.org gerrit flush-caches --cache web_sessions
cobalt.wikimedia.org there is
/root/unfirewall.sh, which will shut off access to Gerrit HTTP(S) and SSH.
Opsen will find them fairly self-explanatory.
Gerrit queue tasks, given you have the appropriate permission you can run either of:
$ ssh -p 29418 gerrit.wikimedia.org ps -w $ ssh -p 29418 gerrit.wikimedia.org gerrit show-queue -w
Tasks can be stuck which starve Gerrit processing. For example people would no more be able to fetch. In such a case, granted you are in the Administrators group you can kill a task by id:
$ ssh -p 29418 gerrit.wikimedia.org kill <SOMEID>
Forcing Replication re-runs
When forcing a replication run through gerrit's replication plugin, stay clear of the
--all switch, as it breaks renamed projects on github.
For example, both gerrit and github have a
mediawiki repository. But github's
mediawiki repository is actually
mediawiki/core in gerrit. As gerrit also has a
mediawiki repository, forcing replication on all projects would push gerrit's
mediawiki repository onto github's
mediawiki repository. Thereby, github's
mediawiki repository falls over, and for examples the release tags from github's
mediawiki are gone (cf. task T100409).
This currently (2015-05-28) affects us for at least for the following repos:
|Gerrit's name||Overloaded name on GitHub|
If you by accident forced replication with
--all, force replication of the effective gerrit repos to make up for it (i.e.:
As of July 2018 replication to github requires a repository to be manually created on github using the naming scheme gerrit expects, i.e.
/ replaced with
Github owners will be able to create these repositories.
Once a repository is created, replication to that repo can be forced using the gerrit replication ssh command.
$ ssh -p 29418 gerrit.wikimedia.org replication start operations/debs/trafficserver --wait
mediawiki-replication must be allowed to read
refs/heads/* for replication to GitHub to work. This is because it is currently set as the
etc inside the Gerrit repository home.
replication_log should be checked in addition to the
gerrit.log for mention of anything that could be wrong with replication.
Any changes to
replication.config should be followed with a restart of the gerrit service to ensure that those changes are active. There is a bug with the replication plugin where it will reload the
replication.config (and acknowledge that the configuration has been reloaded in the logs), but actually operate with a broken configuration in an unknown state until a restart.
ORES mirroring setup is dangerous and unusual, but should, nevertheless, be documented.
The canonical repository for ORES lives on github. This is repository is observed by diffusion. This can done in the
Manage Repository section of diffusion code browser. From there, phabricator "Mirrors" that repository (also available in diffusion management settings) to gerrit.
This setup is discouraged.
How to setup Gerrit in a Cloud VPS project
JavaMelody monitors the state of the Gerrit JVM.
- JavaMelody in Gerrit (only accesible to logged-in Gerrit Administrators/Gerrit Managers)
- Grafana Gerrit JavaMelody Dashboard
- Memory Usage
- GC Timing
- Grafana GC Timing
- Garbage collection metrics. Times in the 100s of milliseconds, rather than in the 10s of milliseconds can be indicative of a problem (running low on memory)
- Active Threads
For users having the | viewCaches capability, various internal Gerrit metrics can be retrieved via:
Which obviously requires authentication. That complements
Logs are available on the gerrit servers at:
/var/log/gerrit/. There are a number of logfiles:
gerrit.log: This is the main log file and will show stacktraces and errors
gerrit.logbug not really human readable. For sending structured logs to logstash.
sshd_log: Log of sshd events
gc_log: Logs for
git gcnot the JVM garbage collection (those logs are available in
plugin_log: Info about plugins being loaded and reloaded, this information is also in
Gerrit sits behind apache. Apache access and error logs are both in
Gerrit's logs are sent to logstash using log4j. You can find its logs by searching with
A thread dump is often useful in troubleshooting. To capture a thread dump use
jstack. This code should be safe to run at any time, and is run frequently while Gerrit is running:
sudo -u gerrit2 jstack -l $(pgrep java) > "/srv/gerrit/jstack-$(date +%Y-%m-%d-%H%M%S).dump"
It's often useful to upload the resulting file to https://fastthread.io/ to detect problems.
This command isn't run very often, unsure how safe it is to run; kept here for folks who are familiar with jstat
Display a summary of garbage collection statistics every 1000 ms:
sudo -u gerrit2 /usr/lib/jvm/java-8-openjdk-amd64/bin/jstat -gcutil "$(pgrep -u gerrit2 java)" 1000
Java heap usage
Requires openjdk-X-dbg for the debugging symbols
sudo /usr/lib/jvm/java-8-openjdk-amd64/bin/jmap -heap "$( pgrep -u gerrit2 java)"