Jenkins is a software coming from the Java community, to assist in building a continuous integration system. It offers a highly modular system which has allowed other community to add their own plugins (such as GIT, PHP...). Wikimedia Foundation is using Jenkins to run CI tests for MediaWiki and various other products and tools, and to generate alpha builds of the Wikipedia Android app.
Upstream does a release every week and a stable one from time to time, also named Long Term Support (LTS). We will certainly want to at least upgrade to LTS whenever it is released and possibly to a snapshot in between depending on bug fixes.
Debian packages can be installed on Ubuntu and are made available at:
cd /srv/wikimedia reprepro checkupdate jessie-wikimedia
verify the output and then
reprepro update jessie-wikimedia
It seems that recent Jenkins repository layout changes have broken the mechanism above by triggering a reprepro bug/missing feature (double redirects). You can always fall back by fetching the LTS deb manually from http://pkg.jenkins-ci.org/debian-stable/ and then run:
reprepro -C thirdparty includedeb jessie-wikimedia jenkins_1.480.3_all.deb reprepro export jessie-wikimedia
Jenkins is deployed on contint1001 and contint2001 hosts, reserved for continuous integration purposes. It can safely be restarted at anytime as long as the developer community is aware about it ( #mediawiki / #wikimedia-dev ). You will most probably want to have one continuous integration person floating around while doing the upgrade since plugin dependencies might break once Jenkins is upgraded.
To upgrade the package, just apt-get update && apt-get upgrade jenkins. (or tell puppet to ensure "latest")
Main application logs:
journalctl -u jenkins -f
Jenkins plugins are placed in /var/lib/jenkins/plugin/ and ends with a .hpi extension. To disable a plugin, rename it to .hpi.disable and restart Jenkins!
Java thread dump
Whenever Jenkins appears to be stuck or facing high CPU usage, you will want to look at the Java threads: https://integration.wikimedia.org/ci/threadDump
You can also send signal 3 to the java process, Jenkins will write a thread dump in /var/log/jenkins/jenkins.log
kill -3 "pid of jenkins"
kill -3 apparently kills Jenkins :(
jstack -F <pid of jenkins>
And there is a more verbose dump written to /var/log/jenkins/jenkins.log
For deadlock detection:
jstack -l -F <pid of jenkins>
sudo -u jenkins jstat -gcutil PID_HERE 1000 3
Issues about upgrade done on April 5th 2012 :
Patch a plugin
If you absolutely need to patch a plugin without an upstream release, that's possible. First, fork the Git repository to Gerrit under integration/jenkinsci/, and then submit your patch as a Gerrit change and merge it.
If you need to, update the version number in
pom.xml, for example
1.103-wmf.1. Then run
mvn install, which should create a
<name>.hpi file somewhere. You can upload that file in the Plugin Manager using the "Advanced" tab. If it tells you to restart Jenkins, kill any long running jobs, and then it should restart shortly. Then try out your newly patched plugin!