Jenkins

From Wikitech
Jump to: navigation, search

Jenkins is a software coming from the Java community, to assist in building a continuous integration system. It offers a highly modular system which has allowed other community to add their own plugins (such as GIT, PHP...). Wikimedia Foundation is using Jenkins to run CI tests for MediaWiki and various other products and tools, and to generate alpha builds of the Wikipedia Android app.

Upgrading

Upstream does a release every week and a stable one from time to time, also named Long Term Support (LTS). We will certainly want to at least upgrade to LTS whenever it is released and possibly to a snapshot in between depending on bug fixes.

Updating

Debian packages can be installed on Ubuntu and are made available at:

http://pkg.jenkins-ci.org/debian/ http://pkg.jenkins-ci.org/debian-stable/

Per ops convention, the packages need to be copied to apt.wikimedia.org. Reprepro is configured to do so semi-automatically, so on install1002 one only needs to do

 cd /srv/wikimedia
 reprepro checkupdate jessie-wikimedia

verify the output and then

reprepro update jessie-wikimedia
Warning Warning: It seems that recent Jenkins repository layout changes have broken the mechanism above by triggering a reprepro bug/missing feature (double redirects). You can always fall back by fetching the LTS deb manually from http://pkg.jenkins-ci.org/debian-stable/ and then run:
 reprepro -C thirdparty includedeb jessie-wikimedia jenkins_1.480.3_all.deb
 reprepro export jessie-wikimedia

Deploying

Jenkins is deployed on contint1001 and contint2001 hosts, reserved for continuous integration purposes. It can safely be restarted at anytime as long as the developer community is aware about it ( #mediawiki / #wikimedia-dev ). You will most probably want to have one continuous integration person floating around while doing the upgrade since plugin dependencies might broke once Jenkins is upgraded.

To upgrade the package, just apt-get update && apt-get upgrade jenkins. (or tell puppet to ensure "latest")

How to

Restart Jenkins

See mw:Continuous integration/Jenkins#Restart.

Logs

Main application logs:

tail -f /var/log/jenkins/jenkins.log

Disable plugin

Jenkins plugins are placed in /var/lib/jenkins/plugin/ and ends with a .hpi extension. To disable a plugin, rename it to .hpi.disable and restart Jenkins!

Java thread dump

Whenever Jenkins appears to be stuck or facing high CPU usage, you will want to look at the Java threads: https://integration.wikimedia.org/ci/threadDump

You can also send signal 3 to the java process, Jenkins will write a thread dump in /var/log/jenkins/jenkins.log

kill -3 "pid of jenkins"

kill -3 apparently kills Jenkins :(


Another way:

jstack -F <pid of jenkins>

And there is a more verbose dump written to /var/log/jenkins/jenkins.log

For deadlock detection:

jstack -l -F <pid of jenkins>

Java info

sudo -u jenkins jstat -gcutil PID_HERE 1000 3

Links

Worklog

Issues about upgrade done on April 5th 2012 :

See also