Nova Resource:Jenkins/Documentation/Archive

From Wikitech

Jenkins/Documentation/Archive

Description

Jenkins setup and Gerrit+Jenkins integration Projects used in production

Ultimate plan

The ultimate plan here is to replace all of our existing Gerrit hooks with Jenkins jobs that are triggered by the Gerrit Trigger Plugin. The OpenStack fork of this plugin can trigger jobs on any event, not just patchset-created. This will not only allow us to integrate Jenkins with Gerrit for the purposes of running tests, but also to move all other Gerrit hooks (like IRC notifications and puppet lint checks) to Jenkins. We have these in Gerrit's hook system currently, but that hook system sucks (has no built-in facility to run different hooks for different repos for example).

List of jobs I'd like to see:

  • lint checks run on patchset-created, post a V:-1 or V:+1 review to Gerrit
    • write checks for PHP, JavaScript, puppet, Python, ... and enable one or more of them on a per-repo basis
    • produce JUnit XML output so each file shows up as a test case in Jenkins
  • IRC notifications for all events
    • in the short term, this putting ircecho on the Jenkins box instead of the Gerrit box
    • in the long term, Ryan wants to set up a message queue for IRC bots
  • unit test runs
    • alternative 1: run these upon patchset-created, post a V:-1 or V:+1 review to Gerrit. Running arbitrary code in Jenkins present security challenges though. We could address these by giving the test jobs resource limits and no network access
    • alternative 2: upon a comment-added with a +2, kick off a test run and have Jenkins submit the change if successful (and -1 it otherwise). This is what the OpenStack people did to solve the security issue
    • OpenStack's Jenkins also runs the tests against the merged state, not the submitted state (i.e. they check out the patchset, try to merge it into HEAD, and run the tests against the merge commit). This verifies that merging the commit won't break the mainline, which is what people actually care about. However, for this to be reliable, this has to be done right before merging, which means alternative 2 must be used (I asked the OpenStack people about race conditions, but apparently Jenkins doesn't run builds in parallel so these can't occur; that would have to be verified though). Alternative 1 can be used in addition if we want, because 1 and 2 aren't mutually exclusive. Generally, every test that we run on patchset-created should also be run pre-merge
  • RT integration: upon changeset-merged, close the relevant RT ticket if any. This is currently in a Gerrit hook written in Python, but should be easy to port to a Jenkins job
  • Bugzilla integration: same as RT integration, except that it hasn't been implemented yet
  • Puppetize all of this
    • What's going on with the Jenkins puppetization, anyway? misc::jenkins provides Jenins from a 3rd-party PPA (which Ryan hates), misc::contint::test::jenkins sets up jobs and Apache proxying but doesn't install the package at all, which means it can't be used to set up new machines
    • I also really want to puppetize this in a generic and reusable way, such that it's trivial to 1) use (most of the) puppet classes for both labs and production and 2) use some of the puppet classes to set up Jenkins in a totally different environment (i.e. have a generic Jenkins puppetization)

Tasks done

  • Installed Jenkins (using misc::jenkins)
  • Installed Gerrit Trigger Plugin
  • Put in Jenkins job for "Gerrit prep" which creates the theoretical merge commit for testing
  • Moved lint job from production Jenkins
  • Rewrote lint job in bash and made it output JUnit's testcase XML format

Tasks to do

  • Rewrite lint job in PHP to use parsekit
  • Port IRC notifications to Jenkins jobs
  • Port RT integration to Jenkins jobs
  • Write lint jobs for puppet, Python, JS
  • Port unit test jobs
  • Implement BZ integration jobs (commenting on or closing bugs in certain cases)

Puppetization notes

  • Installing the OpenStack fork of the Gerrit Trigger Plugin
  • Gerrit plugin needs git-core
  • git needs to be configured with git config --global user.email ... (and user.name too)
  • Lint job needs php5-cli
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Jenkins/Documentation/Archive&oldid=4313"