Jenkins is software from the Java community, that assists in building a continuous integration system. It offers a highly modular system which has allowed other communities to add their own plugins (such as GIT, PHP...). Wikimedia Foundation is using Jenkins to run CI tests for MediaWiki and various other products and tools, and to generate alpha builds of the Wikipedia Android app.
Release engineering maintains four Jenkins instances:
|contint1001.wikimedia.org||The CI Jenkins https://integration.wikimedia.org/ci/|
|contint2001.wikimedia.org||Cold CI Jenkins (stopped)|
|releases1001.eqiad.wmnet||The releasing Jenkins https://releases-jenkins.wikimedia.org/|
|releases2001.codfw.wmnet||Cold release Jenkins|
To get administrative access on the CI Jenkins instance you would need to be added to LDAP group
ciadmin. As of December 2019, the releases Jenkins permissions are managed differently (ask release engineering team).
To get shell access, your account has to be added to the
releasers-mediawiki shell groups. That is done in operations/puppet
modules/admin/data/data.yaml. One can use
matrix.py to list their group:
$ ./matrix.py hashar grp/users hashar contint-admins OK contint-docker OK contint-roots OK deployment OK gerrit-deployers OK gerrit-root OK labnet-users OK releasers-mediawiki OK
If one needs root access (to look at Apache logs, upgrade Jenkins, change file permissions etc), that is granted by the group
contint-roots for both CI and release Jenkins hosts.
Upstream does a release every week and a stable one from time to time, also named Long Term Support (LTS). We will certainly want to at least upgrade to LTS whenever it is released and possibly to a snapshot in between depending on bug fixes.
Get the package
Debian packages can be installed on Ubuntu and are made available at:
cd /srv/wikimedia reprepro checkupdate jessie-wikimedia reprepro checkupdate stretch-wikimedia
verify the output and then
reprepro update jessie-wikimedia reprepro update stretch-wikimedia
Review the upstream changelog https://jenkins.io/changelog-stable/ and try to guess what might cause havoc. There is no clear science there. The community reported bugs might raise a red flag.
Jenkins is deployed on contint1001 and contint2001 hosts, reserved for continuous integration purposes. It can safely be restarted at anytime as long as the developer community is aware about it (!log it in #wikimedia-releng and announce it in #wikimedia-operations NNN-FIXME minutes beforehand).
When doing an upgrade, you will most probably want to have one continuous integration person floating around because plugin dependencies might break once Jenkins is upgraded.
We usually upgrade the releases hosts first since they are less impactful if something goes south. Then the CI ones.
When Jenkins is restarted, all currently jobs are aborted. They will be automatically enqueued again by Zuul when Jenkins is back.
To upgrade the package, just
apt-get update && apt-get upgrade jenkins (requires
A few days before releasing a security update, upstream sends a pre-announcement to a mailing list (jenkinsci-advisories). The internal mailing list of the release engineering team is subscribed to it. Upon reception, a security task is filled in Phabricator against
The are two kind of security updates for Jenkins:
- an issue in Jenkins itself, which would triggers a release of Jenkins. It is thus important to always stay up to date and follow Jenkins LTS cycle.
- issues in plugins, they typically affect several plugins which are released in bulk
Jenkins security update
When the security update is made public, one has to first review the security bulletin to gauge how affected we are. Some security release of Jenkins do not affect us at all, some others are absolutely critical and require an immediate upgrade. The upgrade is as above: The Jenkins Debian package is updated using reprepro as described above and
apt install it.
Plugins security update
For plugins, the update is done via the plugin manager. Use the
Check now button at the bottom of the page to fetch the latest list of plugins from upstream, then update affected plugins. Note: plugins might have back compatibility issue or could expect XML elements different than those generated by Jenkins Job Builder, there is no solid process there.
Main application logs:
journalctl -u jenkins -f
Jenkins plugins are placed in
/var/lib/jenkins/plugin/ and ends with a
.hpi extension. To disable a plugin, rename it to
.hpi.disable and restart Jenkins!
Java thread dump
Whenever Jenkins appears to be stuck or facing high CPU usage, you will want to look at the Java threads: https://integration.wikimedia.org/ci/threadDump
You can also send signal 3 to the java process, Jenkins will write a thread dump in
kill -3 "pid of jenkins"
kill -3 apparently kills Jenkins :(
jstack -F <pid of jenkins>
And there is a more verbose dump written to /var/log/jenkins/jenkins.log
For deadlock detection:
jstack -l -F <pid of jenkins>
sudo -u jenkins jstat -gcutil PID_HERE 1000 3
Patch a plugin
If you absolutely need to patch a plugin without an upstream release, that's possible. First, fork the Git repository to Gerrit under integration/jenkinsci/, and then submit your patch as a Gerrit change and merge it.
If you need to, update the version number in
pom.xml, for example
1.103-wmf.1. Then run
mvn install, which should create a
<name>.hpi file somewhere. You can upload that file in the Plugin Manager using the "Advanced" tab. If it tells you to restart Jenkins, kill any long running jobs, and then it should restart shortly. Then try out your newly patched plugin!