Gerrit
We use Gerrit for our code review system for Git.
Quick links
- Gerrit
- Built-in Gerrit documentation
- Gerrit test install, hosted in devtools Cloud VPS project
- Commit message guidelines
- Gerrit JavaMelody Monitoring
- Gerrit test instance
- Monitoring dashboard: https://grafana.wikimedia.org/dashboards/f/5AnaHr2Mk/gerrit
Details
Gerrit is installed on gerrit1001 in the prefix /var/lib/gerrit2 and serves public request at https://gerrit.wikimedia.org/r/) . A replica runs at gerrit2001 which serves requests at https://gerrit-replica.wikimedia.org/r/ ), it is intended for a switch over and for various bots in order to offload the primary instance.
Gerrit is a Java daemon which listens for HTTP connections (port 8080) and SSH connections (port 29418). Apache proxies the relevant URLs on port 80 and 443 through to Gerrit. The SSH port provides a restricted shell for Git checkouts and administrative commands.
Since the Gerrit v3.2 upgrade in summer 2020, Gerrit no longer uses a conventional, relational database. So if you read somewhere about Gerrit's MySQL (or similar) database or "reviewdb", it is stale information. Instead of a relational database, Gerrit now stores the needed data directly in the git repositories (NoteDB) and to speed up lookup it creates indices to speed up access, also known has secondary index. These indices are Lucene indices and H2 (flat file database engine from the Java world).
Git repositories are stored at /srv/gerrit/git/
.
Gerrit uses the LDAP instance shared with Wikimedia Cloud Services for authentication. Accounts under ou=people,dc=wikimedia,dc=org
and groups under ou=groups,dc=wikimedia,dc=org
are exposed to Gerrit. The Gerrit account name is the cn
field in LDAP. In order to log in to Gerrit, a user needs to already have a Wikimedia developer account. See Help:Getting Started for the process of getting a developer account.
To find out what version of Gerrit is running, you can use either of the following two commands:
java -jar /var/lib/gerrit2/review_site/bin/gerrit.war version ssh -p 29418 gerrit.wikimedia.org gerrit version
As of November 2020, Gerrit runs with Java 8 (upgrade to Java 11 is task T268225).
Using git
See:
- Gerrit online documentation
- Our own documentations:
Alternatively, you can follow the Git & Gerrit codelab.
Upgrade procedures
Since Feb 2nd 2021, we use the upstream war file fetched from https://www.gerritcodereview.com/ but we still build plugins from sources. The source code is held in the Gerrit repository operations/software/gerrit
.
Prerequisites
- Install OpenJDK 8 and git (
openjdk-8-jdk
andgit
packages on Debian) - Bazel is required (as of July 2020 at least bazel 3.4.1, can be checked in
.bazelversion
).
Instead of installing Bazel directly, consider installing Bazelisk, which is a Bazel frontend that automatically installs the needed Bazel version for the Gerrit branch you are working on. Bazelisk saves you a lot of trouble when building Gerrit, so if you can, use it.
Update our repository
- Clone gerrit and submodules and add upstream as a remoteOur repository HEAD points to the branch holding the material for the deployment
git clone https://gerrit.wikimedia.org/r/operations/software/gerrit cd gerrit git remote add upstream https://gerrit.googlesource.com/gerrit
deploy/wmf/stable-3.2
- Checkout the
stable-3.2
branch (as of November 2020) that tracks the current upstream release:git checkout stable-3.2 origin/stable-3.2
- Fetch the upstream branch and fast-forward our copy of their branch to the latest tag and push the
stable-3.2
directly to Gerrit:git fetch --tags upstream/stable-3.2 git merge --ff-only v3.2.3 git push --tags origin HEAD:stable-3.2
- Checkout
wmf/stable-3.2
and merge the latest tag, push the resulting SINGLE merge commit up for reviewgit checkout wmf/stable-3.2 git merge v3.2.7
Next update git submodules. The .gitmodules
file comes from upstream and uses relative urls. Since your work copy points to our Gerrit, the submodules would be fetched from our Gerrit. Instead we have to instruct git to rewrite the URLs to use upstream repositories. This can be done using the git setting url.<base>.insteadOf = <our url>
:
- Update git submodules from the upstream repository:
git -c url."https://gerrit.googlesource.com".insteadOf="$(dirname $(git config --get remote.origin.url))" submodule update --init
- Push for review. At this point, you will likely be ahead of
wmf/stable-3.2
by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:git push origin HEAD:refs/for/wmf/stable-3.2%topic=v3.2.3
Fetch our additional plugins
As our Gerrit instance runs some plugins which are not part of Gerrit upstream set of plugins, we need to fetch them. Some plugins support out-of-tree builds, but the most stable approach is to simply clone them into directories underneath Gerrit's plugins
directory in a checkout of our wmf/stable-3.2
branch. You can use the following short bash script (which assumes you can ssh
to our Gerrit and you have got jq
installed) to clone the needed repositories:
#!/bin/bash
set -e
set -o pipefail
INSTALLED_PLUGINS="$(ssh -p 29418 gerrit.wikimedia.org gerrit plugin ls --format JSON | jq --raw-output 'keys | .[]')"
DEPENDENCIES_FOR_BUILDING="its-base"
cd plugins
for PLUGIN in $INSTALLED_PLUGINS $DEPENDENCIES_FOR_BUILDING
do
if [ ! -e "$PLUGIN" ]
then
echo "Cloning $PLUGIN ..."
git clone "https://gerrit.googlesource.com/plugins/$PLUGIN"
fi
done
echo "Success. All needed plugins cloned."
Knowing which branch to choose for which repository is not so straight forward. If a plugin (most don't) provide version tags, use them if you are building Gerrit for a specific version. Otherwise, many plugins come with stable-
branches. Use the one for the Gerrit version you are building. If a plugin does not offer any of these, try the master
branch.
Please go through each of the plugins individually to find the tag/branch that best fits your desired target. There are only 20 or so plugins, so while it's annoying, it will not take ages.
The following script go through each of our installed plugins, attempts to checkout the tag, else the stable branch and finally fallback to master:
#!/bin/bash
set -eu -o pipefail
INSTALLED_PLUGINS="$(ssh -p 29418 gerrit.wikimedia.org gerrit plugin ls --format JSON | jq --raw-output 'keys | .[]')"
echo "Checkout tag or branch or master..."
for PLUGIN in $INSTALLED_PLUGINS; do
(
cd "plugins/$PLUGIN"
git checkout v3.2.7 || git checkout stable-3.2 || git checkout master
);
done;
echo "Done."
echo "Show the resolved branch or tag if HEAD is detached"
for PLUGIN in $INSTALLED_PLUGINS; do
(
cd "plugins/$PLUGIN"
echo "$PLUGIN"
git describe --all
git branch --points-at=HEAD
)
done;
echo "Done."
Fix-up of gitiles plugin
The gitiles
plugin per default fetches fonts directly from Google. Since that's a third party dependency that we do not want, we instead want gitiles to use the fonts that Gerrit itself offers.
To achieve that, make sure your gitiles code includes change I4f4a0b7dd575cbc27641063e05d13b8a43a51d8b. The change did not get much traction upstream, so it probably won't land there. We nonetheless want to include that change.
Fix-up of javamelody plugin
The javamelody
plugin comes with external dependencies. Gerrit's build system does not automatically pick them up. To allow Gerrit to build the plugin, merge the contents of plugins/javamelody/external_plugin_deps.bzl
into Gerrit's own plugins/external_plugin_deps.bzl
Most plugins come with build instructions usually available at src/main/resources/Documentation/build.html
.
Build Gerrit
Gerrit core
As of Feb 2nd 2021 we no more build Gerrit ourselves, we instead use upstream .war.
- Checkout the
wmf/stable-3.2
(as of November 2020)git checkout -b wmf/stable-3.2 origin/wmf/stable-3.2
Production uses Java 8 as of November 19th 2020 which we will eventually switch to Java 11 ( https://phabricator.wikimedia.org/T268225 ). Meantime, Bazel has to be pointed to your local Java 8 installation:
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
.
Build thebazel-bin/release.war
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 bazel build release
Note that if your building Gerrit v3.3 (or later) or master from 2020-10-06 (or later) and you're targeting Java 8, you'll also have to add the argument
--java_toolchain=//tools:error_prone_warnings_toolchain
(this is untested, but claimed on the gerrit mailing list).
Plugins
- Build the plugins
jar
files:for PLUGIN in $(ssh gerrit.wikimedia.org gerrit plugin ls --format JSON | jq --raw-output 'keys | .[]') ; do bazel build --workspace_status_command="python ./tools/workspace_status_release.py" plugins/$PLUGIN ; done
Troubleshooting
Current Bazel version is 0.18.1, expected at least 0.19.0
- Bazel updates all the time, update your bazel version from the bazel website (ensure you verify the version signature with the [ https://bazel.build/bazel-release.pub.gpg bazel-release.pub.gpg] key. Or even better: Use Bazelisk, which is a transparent wrapper for Bazel that takes care of preparing the correct version for you.
curl: (35) error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
- bazel shells out to curl to download things and, sadly,
repo.eclipse.org
only supports TLSv1openssl s_client -connect repo.eclipse.org:443 -tls1
. You can modify your openssl settings in/etc/ssl/openssl.cnf
. If yourMinProtocol
is set toTLSv1.2
(because you're awesome) — change it toTLSv1
(because not everyone is awesome). Be sure to change it back when you're done.
Upload artifacts
- Copy the resulting
bazel-bin/release.war
andbazel-bin/plugins/$PLUGIN/$PLUGIN.jar
to somewhere safe - Checkout the
deploy/wmf/stable-X
branch that aligns with the build branch Copy in therelease.war
you saved as./gerrit.war
- Copy all the plugin jars over to
./plugins
in the deploy branch - Commit the changes, upload to Gerrit, etc
- Run
./upload_artifacts.py --version=2.15.2 gerrit.war plugins/foo.jar ...
for the core application and all updated plugins (cf: Archiva, specifically setting up your~/.m2/settings.xml
- Note: Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins so I wrote the tool above
Deploying
- SSH to the deployment master
- Navigate to
/srv/deployment/gerrit/gerrit
- Fetch & checkout the appropriate
deploy/wmf/stable-X
branch - Deploy to gerrit2001 to ensure that there are no errors with git-fat et al
scap deploy -l gerrit2001.wikimedia.org 'Gerrit to [version] on gerrit2001'
- Deploy to gerrit1001
scap deploy -l gerrit1001.wikimedia.org 'Gerrit to [version] on gerrit1001'
- If you're only deploying plugins, you're done, otherwise SSH to the Gerrit master and issue
sudo service gerrit restart
Restarting
If you need to restart Gerrit (e.g. because it is OOMing), run sudo systemctl restart gerrit
. This will take a few seconds before it comes back.
Migrating
Example migration ticket (cobalt -> gerrit1001): https://phabricator.wikimedia.org/T222391
Topic branch from gerrit1001 migration: https://gerrit.wikimedia.org/r/q/topic:%22gerrit1001%22+(status:open%20OR%20status:merged)
Follow these steps to migrate from one server to another:
Make sure you run rsync with --delete
- Rsync /srv/gerrit/git/ , /srv/gerrit/plugins and /var/lib/gerrit2/review_site/ from <old_host> to <new_host>
rsync --archive --verbose --delete /srv/gerrit/git/ rsync://<new_host>.wikimedia.org/gerrit-data/git/
rsync --archive --verbose --delete /srv/gerrit/plugins/ rsync://<new_host>.wikimedia.org/gerrit-data/plugins/
rsync --archive --verbose --delete /var/lib/gerrit2/review_site/ rsync://<new_host>.wikimedia.org/gerrit-var-lib/
(Note: We are using rsync protocol directly and 'gerrit-data' and 'gerrit-var-'lib' are the names of rsync modules, not file system pathes. They are defined in "class profile::gerrit::migration".)
- Stop gerrit && disable puppet on <new_host>
- Create something similar to https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/535966/
- Stop puppet on <old_host> + <replica>
- Create something similar to https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/541110/
- Create something similar to https://gerrit.wikimedia.org/r/#/c/operations/dns/+/541111/
- Stop gerrit on <old_host>
- Repeat the rsync commands above.
- Rename /var/lib/gerrit2/review_site/data/javamelody/r_<old_host> to /var/lib/gerrit2/review_site/data/javamelody/r_<new_host> on <new_host>.
- Run puppet on <new_host> + <old_host>
- Start gerrit on <new_host>
- Hack DNS authdns-update to clone from gerrit-replica temporarily, deploy DNS change
- Manually copy apache2 site config for gerrit.wm.org with scp from <old_host> to <new_host>, restart apache
- Manually run command from list_mediawiki_extensions cron to create /var/www/mediawiki-extensions.txt
- Run the online reindexer
- Decom <old_host> (create a ticket like https://phabricator.wikimedia.org/T236187)
Creating new repositories
See mw:Git/Creating new repositories. Those without administrative access to Gerrit should use mw:Gerrit/New repositories instead.
Inspecting an account settings
Since Gerrit 2.15, accounts are stored in All-Users.git
. One first need to find the account reference, that can be done by checking which references a user can read on All-Users.git:
$ ssh -p 29418 gerrit.wikimedia.org gerrit ls-user-refs --project All-Users --user 'johndoe@wikimedia.org'
refs/meta/config
refs/users/34/1234
refs/users/self
$
Note: for accounts having extended permissions on All-Users.git
, that will show all references which is not useful.
Then on the server side (or remotely if you have privileges), you can list the configuration files and dump their content:
$ cd /srv/gerrit/git/All-Users.git
$ git ls-tree refs/users/34/1234
100644 blob xxx account.config
100644 blob yyy authorized_keys
100644 blob zzz preferences.confg
100644 blob ddd watch.config
$ git show refs/users/34/1234:authorized_keys
ssh-rsa AAAAxyz... RSA-1024
The All-Users
database can be browsed locally via git. The easiest way to find a user given the All-Users
git repo is to git-grep for the username or email in the refs/meta/external-ids
ref:
$ mkdir All-Users && cd All-Users && git init
$ git pull https://gerrit.wikimedia.org/r/All-Users refs/meta/external-ids
$ git grep johndoe
Become an Administrator
Gerrit management is granted by the Administrators group which includes the gerrit2
system user and the LDAP group gerritadmin
. To gain Administrator right, one thus has to be added in LDAP.
Being an administrator grants all global capabilities as described at Gerrit Access Control - global capabilities. It is essentially the same as having root access on Gerrit.
The LDAP groups membership is cached in Gerrit for 1 hour (as of January 2020). One thus has to wait for the cache to expire or have an administrator to flush the cache:
ssh -p 29418 gerrit.wikimedia.org gerrit flush-caches --cache ldap_groups
If right is given as a one off, one would most probably want to be removed from the group once the rights are no more required.
Disabling / Blocking an account
Disable User
This action is limited to Gerrit administrators.
If an account needs to be disabled for some reason, there's two steps to complete:
- remove the ssh keys
- set the account to inactive
To set an account inactive, use this command:
$ ssh gerrit.wikimedia.org -p 29418 gerrit set-account --inactive <Username here>
To enable the user again:
$ ssh gerrit.wikimedia.org -p 29418 gerrit set-account --active <Username or user id here>
If you get an error that the user can't be found, try using the user id instead (can be obtained by grepping the all-users database).
And flush the accounts cache on the replica:
$ ssh gerrit-replica.wikimedia.org -p 29418 gerrit flush-caches --cache accounts
Flush all sessions
![]() | Do not flush all sessions when banning a single user. As of May 2019 setting a user to inactive should invalidate their session automagically |
Flushing web_sessions
caches makes everyone re-login (aka, the sledgehammer). It should not be necessary in most circumstances; however, the command is still in this documentation in case it is needed in an unforeseen situation:
$ ssh -p 29418 gerrit.wikimedia.org gerrit flush-caches --cache web_sessions
Firewalling Gerrit
On gerrit1001.wikimedia.org
there is /root/firewall.sh
and /root/unfirewall.sh
, which will shut off access to Gerrit HTTP(S) and SSH.
Opsen will find them fairly self-explanatory.
Tasks management
Gerrit queue tasks, given you have the appropriate permission you can run either of:
$ ssh -p 29418 gerrit.wikimedia.org ps -w $ ssh -p 29418 gerrit.wikimedia.org gerrit show-queue -w
Tasks can be stuck which starve Gerrit processing. For example people would no more be able to fetch. In such a case, granted you are in the Administrators group you can kill a task by id:
$ ssh -p 29418 gerrit.wikimedia.org kill <SOMEID>
Replication
Forcing Replication re-runs
When forcing a replication run through gerrit's replication plugin, stay clear of the --all
switch, as it breaks renamed projects on github.
For example, both gerrit and github have a mediawiki
repository. But github's mediawiki
repository is actually mediawiki/core
in gerrit. As gerrit also has a mediawiki
repository, forcing replication on all projects would push gerrit's mediawiki
repository onto github's mediawiki
repository. Thereby, github's mediawiki
repository falls over, and for examples the release tags from github's mediawiki
are gone (cf. task T100409).
This currently (2015-05-28) affects us for at least for the following repos:
Gerrit's name | Overloaded name on GitHub |
---|---|
mediawiki/core
|
mediawiki
|
VisualEditor/VisualEditor
|
VisualEditor
|
oojs/core
|
oojs
|
If you by accident forced replication with --all
, force replication of the effective gerrit repos to make up for it (i.e.: mediawiki/core
, VisualEditor/VisualEditor
, ...)
Github
As of July 2018 replication to github requires a repository to be manually created on github using the naming scheme gerrit expects, i.e. /
replaced with -
(FrEx: operations/debs/trafficserver
-> operations-debs-trafficserver
.
Github owners will be able to create these repositories.
Once a repository is created, replication to that repo can be forced using the gerrit replication ssh command.
$ ssh -p 29418 gerrit.wikimedia.org replication start operations/debs/trafficserver --wait
Troubleshooting
The group mediawiki-replication
must be allowed to read refs/tags/*
and refs/heads/*
for replication to GitHub to work. This is because it is currently set as the authGroup
in replication.config
inside etc
inside the Gerrit repository home.
Additionally, the replication_log
should be checked in addition to the gerrit.log
for mention of anything that could be wrong with replication.
Any changes to replication.config
should be followed with a restart of the gerrit service to ensure that those changes are active. There is a bug with the replication plugin where it will reload the replication.config
(and acknowledge that the configuration has been reloaded in the logs), but actually operate with a broken configuration in an unknown state until a restart.
ORES
ORES mirroring setup is dangerous and unusual, but should, nevertheless, be documented.
The canonical repository for ORES lives on github. This is repository is observed by diffusion. This can done in the Manage Repository
section of diffusion code browser. From there, phabricator "Mirrors" that repository (also available in diffusion management settings) to gerrit.
This setup is discouraged.
How to setup Gerrit in a Cloud VPS project
See Gerrit/Running in Cloud VPS.
Monitoring
JavaMelody monitors the state of the Gerrit JVM. They are collected by Prometheus from https://gerrit.wikimedia.org/r/monitoring?prometheus
- JavaMelody in Gerrit (only accesible to logged-in Gerrit Administrators/Gerrit Managers)
- Grafana "Gerrit" folder"
Important Graphs
- Memory Usage
- GC Timing
- Grafana GC Timing
- Garbage collection metrics. Times in the 100s of milliseconds, rather than in the 10s of milliseconds can be indicative of a problem (running low on memory)
- Active Threads
- Gerrit Active Threads
- Grafana Active Threads
- Usually there are less than 20 active threads at any given time — more than that typically means that you should take a Thread Dump and restart.
Gerrit metrics
On top of the JavaMelody data, Gerrit has internal metrics.
For users having the | viewCaches or View Metrics capabilities, various internal Gerrit metrics can be retrieved via:
Which obviously requires authentication. That complements gerrit show-caches
.
We use the metrics-reporter-prometheus plugin which exposes collected by Prometheus from https://gerrit.wikimedia.org/r/plugins/metrics-reporter-prometheus/metrics . Those Gerrit metrics can also be seen on the JavaMelody MBeans page under the metrics
branch.
See Gerrit Grafana dashboards folder.
Logs
Logs are available on the gerrit servers at: /var/log/gerrit/
. There are a number of logfiles:
gerrit.log
: This is the main log file and will show stacktraces and errorsgerrit.json
: Likegerrit.log
bug not really human readable. For sending structured logs to logstash.sshd_log
: Log of sshd eventsgc_log
: Logs forgit gc
not the JVM garbage collection (those logs are available in/srv/gerrit/jvmlogs
)plugin_log
: Info about plugins being loaded and reloaded, this information is also ingerrit.log
HTTP Logs
Gerrit sits behind apache. Apache access and error logs are both in /var/log/apache2
:
gerrit.wikimedia.org.https.access.log
gerrit.wikimedia.org.https.error.log
Logstash
Gerrit's logs are sent to logstash using log4j. You can find its logs by searching with type:log4j
.
Thread Dump
A thread dump is often useful in troubleshooting. To capture a thread dump use jstack
. This code should be safe to run at any time, and is run frequently while Gerrit is running:
sudo -u gerrit2 jstack -l $(pgrep java) > "/srv/gerrit/jstack-$(date +%Y-%m-%d-%H%M%S).dump"
It's often useful to upload the resulting file to https://fastthread.io/ to detect problems.
Java trace
![]() | This command isn't run very often, unsure how safe it is to run; kept here for folks who are familiar with jstat |
Display a summary of garbage collection statistics every 1000 ms:
sudo -u gerrit2 /usr/lib/jvm/java-8-openjdk-amd64/bin/jstat -gcutil "$(pgrep -u gerrit2 java)" 1000
Java heap usage
Requires openjdk-X-dbg for the debugging symbols
sudo /usr/lib/jvm/java-8-openjdk-amd64/bin/jmap -heap "$( pgrep -u gerrit2 java)"
Troubleshooting / upgrade FAQ
Frequently asked questions regarding issues since the recent Gerrit version upgrade to 3.2.
Plugin install error: TypeError: self.onAction is not a function
If you see this error please hard refresh your browser cache.
For some the Ctrl + F5 or Stift + Ctrl + F5 (or whatever combination your browser wants) is not sufficient.
If you still run into the issue after pressing your browsers F5 combination, please try clearing your caches for good.
For example on Firefox follow steps 1-6 from
If you use a different browser, it should allow you to clear cached web content somewhere in the setting too. Please find that in your browser's help pages and follow the instructions.
Can't upload any patches with git review
If you use git-review to submit patches to gerrit, make sure you have 1.27 or newer (not 1.26). git-review 1.26 does not work with the new gerrit version.