Gerrit

From Wikitech
Jump to: navigation, search

We use Gerrit for our code review system for Git. It supersedes the CodeReview MediaWiki extension we have used for Subversion.

Quick links

Details

Gerrit is installed on cobalt in the prefix /var/lib/gerrit2. We appear to have no slave instances since formey was decommissioned. (Check this!) Replication for read-only is done to contint1001 (continuous integration servers).

It is a Java daemon which listens for HTTP connections (port 8080) and SSH connections (port 29418). Apache on lead proxies the relevant URLs on port 80 and 443 through to Gerrit. The SSH port provides a restricted shell for Git checkouts and administrative commands.

It uses the m2 DB cluster, database name "reviewdb", for most storage. Git repos are on NFS and stored at /var/lib/gerrit2/review_site/git.

It uses the LDAP instance shared with Wikimedia Cloud Services and various other things, for authentication. Accounts under ou=people,dc=wikimedia,dc=org and groups under ou=groups,dc=wikimedia,dc=org are exposed to Gerrit. The Gerrit account name is the "cn" field in LDAP. In order to log in to Gerrit, a user needs to already have a Wikimedia developer account. See Help:Getting Started for the process of getting a developer account.

To find out what version of Gerrit is running, you can use either of the following two commands: 

java -jar /var/lib/gerrit2/gerrit.war version
ssh -p 29418 gerrit.wikimedia.org gerrit version

It is 2.13.8-11-gde96955fb2 at the time of writing.

Using git

See mw:Gerrit/Tutorial and mw:Gerrit/Advanced usage.

Database install

Database installation is handled by Gerrit installation process, you simply need a user named 'gerrit2' with appropriate privileges. Make sure to set the charset of the database to utf8.

Replace <PASSWORD> below and apply: 

 CREATE USER 'gerrit2'@'localhost' IDENTIFIED BY <PASSWORD>;
 CREATE DATABASE reviewdb;
 ALTER DATABASE reviewdb charset=utf8;
 GRANT ALL ON reviewdb.* TO 'gerrit2'@'localhost';
 FLUSH PRIVILEGES;


Upgrading Gerrit

  1. Back up the database (reviewdb on m2) and git directories (/var/lib/gerrit2/review_site/git/). Just In Case.
  2. Download the latest .war file from Gerrit Code Review to /var/lib/gerrit2/.
  3. Stop the gerrit daemon (/var/lib/gerrit2/review_site/bin/gerrit.sh stop)
  4. Start gerrit with the new *.war file using the "init" flag. For example: cd /var/lib/gerrit2 && java -jar gerrit-2.3.war init -d review_site
  5. The updater is going to ask you some questions, keep the current values. When it asks you if you want to copy the *.war file to the bin directory, say yes.
  6. Run puppet (this will restart gerrit)
  7. Profit!!

Note: You may need to restart gerrit after the upgrade is finished.

Getting the war

We build Gerrit + our plugins from master daily with Jenkins (plus when Chad feels like it). These builds are available from integration.mediawiki.org.

Generally speaking, the latest build is what we're running on our Cloud VPS testing project. This will be the case for production soon too (right now we're using a custom 2.4.2 build).

Building the deb

  1. ssh boron.eqiad.wmnet (boron has the role::package::builder so it is the general build host)
  2. git clone https://gerrit.wikimedia.org/r/operations/debs/gerrit
  3. cd gerrit
  4. DIST=jessie-wikimedia pdebuild
  5. cd /var/cache/pbuilder/result/jessie-amd64/
  6. ls gerrit*

Uploading the deb to apt.wikimedia.org

  1. scp <user>@boron.eqiad.wmnet:/var/cache/pbuilder/result/jessie-amd64/gerrit_*wmf.5* . (get files from build host. replace <user> with your user, adjust version string at the end)
  2. scp gerrit_2.13.8+git1-wmf.5* root@install1002.wikimedia.org:/srv/wikimedia/incoming/ (upload all files to install/APT server)
  3. ssh install1002.wikimedia.org
  4. sudo -i (so that reprepro sees the key)
  5. cd /srv/wikimedia/incoming
  6. reprepro -C main include jessie-wikimedia gerrit_2.13.8+git1-wmf.5_amd64.changes (the .changes file, not the .deb)
  7. reprepo ls gerrit
  8. (on another server): apt-get update; apt-get -s install gerrit (-s for simulated). to confirm it's installable
  9. (to make it available on stretch as well): reprepro copy stretch-wikimedia jessie-wikimedia gerrit

Creating new repositories

See mw:Git/Creating new repositories. Those without administrative access to Gerrit should use mw:Gerrit/New repositories instead.

Disabling / Blocking an account

This action is limited to users in the ldap/ops or capability-access-database groups.

If an account needs to be disabled for some reason, there's three steps to complete:

  • remove the ssh keys
  • set the account to inactive
  • make all Gerrit users re-login

The account_id can be found in the accounts table, if you don't know it. 

$ ssh -p 29418 gerrit.wikimedia.org gerrit gsql
gerrit> update accounts set inactive = 'Y' where account_id = 1234;
gerrit> delete from account_ssh_keys where account_id = 1234;

Then flush the web_sessions caches to make everyone re-login (aka, the sledgehammer to log one user out): 

$ ssh -p 29418 gerrit.wikimedia.org gerrit flush-caches --cache web_sessions

Instead of setting account inactive via gsql you can alternatively use this command: 

$ssh gerrit.wikimedia.org -p 29418 gerrit set-account --inactive <Username here>

Tasks management

Gerrit queue tasks, given you have the appropriate permission you can run either of: 

$ ssh -p 29418 gerrit.wikimedia.org ps -w
$ ssh -p 29418 gerrit.wikimedia.org gerrit show-queue -w

Tasks can be stuck which starve Gerrit processing. For example people would no more be able to fetch. In such a case, granted you are in the Administrators group you can kill a task by id: 

$ ssh -p 29418 gerrit.wikimedia.org kill <SOMEID>

Forcing Replication re-runs

When forcing a replication run through gerrit's replication plugin, stay clear of the --all switch, as it breaks renamed projects on github.

For example, both gerrit and github have a mediawiki repository. But github's mediawiki repository is actually mediawiki/core in gerrit. As gerrit also has a mediawiki repository, forcing replication on all projects would push gerrit's mediawiki repository onto github's mediawiki repository. Thereby, github's mediawiki repository falls over, and for examples the release tags from github's mediawiki are gone (cf. task T100409).

This currently (2015-05-28) affects us for at least for the following repos:

Gerrit's name Overloaded name on GitHub
mediawiki/core mediawiki
VisualEditor/VisualEditor VisualEditor
oojs/core oojs

If you by accident forced replication with --all, force replication of the effective gerrit repos to make up for it (i.e.: mediawiki/core, VisualEditor/VisualEditor, ...)

Fixing replication errors due to missing commits

If you see replication jobs in gerrit's queue that have an increasing retry count (>5); for example like here 

 _________________________________________________________________
 christian@spencer // jobs: 0 // time: 15:00:42 // exit code: 0
 cwd: ~/tmp/gerrit-replication-fixup
 ssh gerrit.wikimedia.org gerrit show-queue -w | grep retry
 dea27401              12:59:49.618      (retry 676) push git@github.com:wikimedia/apps-ios-wikipedia

it typically means someone pushed a commit directly to github, by-passing gerrit and code-review. This is not a operational issue for gerrit as replication of other commits just continues and new commits make it into github. It's just that it keeps gerrit unnecessarily busy by retrying the same thing again and again and failing each time. And it pollutes gerrit logs.

This replication issue is fixed with upstream gerrit. Upgrading to >=2.9 should fix it for us. Since we're below 2.9, we have to fix manually; we have to manually bring the commit from github to gerrit.

First, we have to find the missing commit. To do so, log into the host thath is running gerrit (on 2015-07-07 that was ytterbium), and tail /var/lib/gerrit2/review_site/logs/error_log. You should see a line like 

 [2015-07-07 12:48:44,586] ERROR com.googlesource.gerrit.plugins.replication.ReplicationQueue : Cannot replicate to git@github.com:wikimedia/apps-ios-wikipedia
 org.eclipse.jgit.errors.TransportException: git@github.com:wikimedia/apps-ios-wikipedia: Missing unknown 6a7595de888c8d04899cc58065fa8682eb844a39

. This means, one needs to bring commit 6a7595de888c8d04899cc58065fa8682eb844a39 from github to gerrit. So essentially:

  • Clone the github repo.
  • Push the commit to an unused branch in gerrit (refs/heads/sandbox/${user} ;-) ).
  • Wait for replication to happen.
  • Remove the unused branch from gerrit again.

Here's the same thing with more details:

  1. For the relevant project, grant Forge Author Identity, and Forge Committer Identity to the Project and Group Administrators group for the refs underneath refs/heads/sandbox/${username}. Otherwise, we cannot push another person's commit to our branch.
  2. _________________________________________________________________
christian@spencer // jobs: 0 // time: 15:03:47 // exit code: 0
cwd: ~/tmp/gerrit-replication-fixup
git clone https://github.com/wikimedia/apps-ios-wikipedia.git
Cloning into 'apps-ios-wikipedia'...
remote: Counting objects: 17403, done.
remote: Total 17403 (delta 0), reused 0 (delta 0), pack-reused 17403
Receiving objects: 100% (17403/17403), 38.19 MiB | 938 KiB/s, done.
Resolving deltas: 100% (11167/11167), done.
Checking out files: 100% (2260/2260), done.

_________________________________________________________________
christian@spencer // jobs: 0 // time: 15:04:38 // exit code: 0
cwd: ~/tmp/gerrit-replication-fixup
cd apps-ios-wikipedia

_________________________________________________________________
christian@spencer // jobs: 0 // time: 15:04:40 // exit code: 0
cwd: ~/tmp/gerrit-replication-fixup/apps-ios-wikipedia
git branch replication-helper 6a7595de888c8d04899cc58065fa8682eb844a39

_________________________________________________________________
christian@spencer // jobs: 0 // time: 15:04:56 // exit code: 1
cwd: ~/tmp/gerrit-replication-fixup/apps-ios-wikipedia
git push ssh://gerrit.wikimedia.org/apps/ios/wikipedia replication-helper:refs/heads/sandbox/qchris/replication-helper
Counting objects: 71, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (20/20), done.
Writing objects: 100% (44/44), 4.61 KiB, done.
Total 44 (delta 32), reused 33 (delta 23)
remote: Resolving deltas: 100% (32/32)
remote: Processing changes: closed: 746, refs: 1, done    
To ssh://gerrit.wikimedia.org/apps/ios/wikipedia
 * [new branch]      replication-helper -> sandbox/qchris/replication-helper
  3. Wait 1 minute, for replication to run the next try.
  4. Check if the replication job vanished from gerrit's queue (Re-run the show-queue command from the beginning of this section). If it did not vanish, the job is blocked on another commit. The error_log will show you the additional commit that you'll need to bring over from github to gerrit.
  5. Drop the sandbox/qchris/replication-helper again from gerrit:
    _________________________________________________________________
christian@spencer // jobs: 0 // time: 15:15:36 // exit code: 1
cwd: ~/tmp/gerrit-replication-fixup/apps-ios-wikipedia
git push ssh://gerrit.wikimedia.org/apps/ios/wikipedia +:refs/heads/sandbox/qchris/replication-helper
remote: Processing changes: refs: 1, done    
To ssh://gerrit.wikimedia.org/apps/ios/wikipedia
 - [deleted]         sandbox/qchris/replication-helper
  6. Drop the identity forging grants from the project config in gerrit again.

How to setup Gerrit in a Cloud VPS project

See Gerrit/Running in Cloud VPS.

Java trace

   sudo -u gerrit2 /usr/lib/jvm/java-8-openjdk-amd64/bin/jstat -gcutil `pidof GerritCodeReview` 1000 3

See also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Gerrit&oldid=1786483"