How to deploy Wikidata code

From Wikitech
Jump to: navigation, search

This page explains procedures for how to make configuration changes, prepare a new Wikidata build (Wikibase + dependencies) and get code updated on Wikidata. See also how to deploy code that is not specific to Wikidata.

Deployment schedule

Wikidata is on the 'deployment train', meaning that Wikidata is updated the same time as MediaWiki core is updated during the general deployment window. A new Wikidata deployment build (Wikibase branch + dependencies) is normally made every two weeks, so that wikidata.org is updated every other week with the train.

Occasionally small bug fixes or config changes are needed outside the normal deployments, normally these are done within a SWAT. Rarely a special deployment window is scheduled for something related to Wikidata.

How to make a new Wikidata build for branches / production deployment

Make a new deployment branch of Wikibase

On command line with latest master of Wikibase, replacing branch name in example with appropriate branch name:

git checkout -b wmf/1.25wmf8
git push origin wmf/1.25wmf8

Note: It is also possible to make a branch of Wikibase using Gerrit interface.

Make a new Wikidata build

The Wikidata build uses wikidata/build-resources as the base for running composer install. To make a build from scratch with this, do:

1) Obtain WikidataBuildResources:

2) Make new branch on gerrit:

3) Then copy the new build into the Wikidata directory:

  • rm -Rf * in Wikidata
  • cp -R ../WikidataBuildResources/* . into the Wikidata directory.
  • then review and submit the changes to the deployment branch of Wikidata in gerrit.

If the contents of https://github.com/wmde/WikidataBuildResources are unchanged since last deployment build, then you can just do:

  • Clone / checkout previous deployment branch of https://phabricator.wikimedia.org/diffusion/EWDA/
  • push this as a new deployment branch to gerrit: git push origin wmf/1.25wmf8
  • Run npm install (so we can use grunt convenience scripts)
  • grunt branch --branchName="wmf/1.25wmf8"
  • git review wmf/1.25wmf8 to submit the new deployment branch to gerrit, so that jenkins runs tests etc. and you can view the diff in gerrit.

4) Update release tools

Document stuff

  • Ahead of time, document important changes and bug fixes that are in the new branch on mw:Wikidata deployment. This helps keep a record of what changed and when, things to announce to the community, and be on look out for any issues after deployment related to those changes.
  • On mw:Wikidata deployment, important to document any deployment notes (e.g. config changes needed).

To help search for things to document, you can search commit messages via git log, with the --grep option:

git log --grep='DEPLOY'
git log --grep='bug'

Update an existing build

  • Cherry pick the relevant change in Wikibase to the appropriate deployment branch (e.g. wmf/1.27.0-wmf.2) and then +2 code review (a deployer needs to do this), if Jenkins approves and it looks sane.
  • Clone Wikidata.git from gerrit
  • Run composer update --prefer-dist -o --ansi --no-dev or if you want to just update a specific component (e.g. wikibase), then composer update --prefer-dist -o --ansi --no-dev wikibase/wikibase. Note: You might need to wait 5 minutes or so for packagist / composer to find the new change in Wikibase.
  • Check the diff and composer.lock file look sane.
  • Submit the change to review: git review wmf/1.27.0-wmf.2 Do not merge until swat or ready to deploy, since a core submodule update is automatically done (, but see exceptions where automatic submodule updates don't work). This might cause trouble or confusion if someone else is deploying something unrelated before us.

Then poke someone (aude, hoo or Amir, or possibly addshore) or sign up for a SWAT deploy to have Wikidata updated.

If the backport involves i18n messages, then ask to have "scap" run which rebuilds localisation cache. If this is not done, then messages may break at a later time. (e.g. when LocalisationUpdate runs, which pulls in / deploys new translations from translatewiki)

For when the deployment actually happens, see #What to do before / during deployment time.

During deployment

  • Make sure all config changes and other todos in the deployment notes section of mw:Wikidata deployment (for the relevant branch) are done.
  • Wikibase etc. automatically gets deployed along with new branches of MediaWiki core at the regularly scheduled core deployment times. See above and mw:Wikidata deployment and Deployments for schedules and details. Mukunda (20after4) or another WMF person will deploy, same time as he/she deploys the new core branch. (Wikidata is a submodule)
  • When the wikis get switched to new version of core + Wikibase / Wikidata build, then check the wikis to be sure things are good.
  • At deployment time, hang out on #wikidata and #wikimedia-operations IRC channel in case problems occur with our stuff.
  • On mw:Wikidata deployment, mark the deployment as done. (for new code deployment to test.wikidata / test2 / wikidata / wikipedias)
  • Inform Lydia and/or Lea that deployment is done and she will handle announcements and communications to the community. (particularly for deployment of new code)

Deploy a hotfix

Hot fixes are critical bug fixes / fix regressions and are to be avoided, if possible, or kept to a minimum.

Security patches

If you find a security issue, create a task on phabricator (marked as security bug)

Any patches for the issue should not be uploaded to gerrit, not even as a draft, because it is not secure. We want the fix to be deployed before publishing the change to gerrit.

A patch can be made using git-format-patch [1] and uploaded to the phabricator task.

Procedures

Deploy a code change to production

First, you need your change to be merged in the master branch. Cherry-pick your change to the relevant wmf/ branch.

Once your change has been merged to the Wikibase deployment branch, you also need to prepare a change to update the Wikidata deployment branch, without merging it

Such deployment should normally occur during the SWAT deploys, windows 3 times per day from Monday to Thursday where you can ask code to be pushed to the production servers.

We normally don't deploy Fridays or week-end. If there is an emergency, you can ask for an exemption on #wikimedia-operations: ping greg-g or absent, RainbowSprinkles. One of these deployers can then deploy your change.

The fix should be checked on test.wikidata and test2.wikipedia.org

Please stay on #wikimedia-operations #wikidata after deployment, to be able to react if there is any issue.

You can point this page to the deployer if they're not used to Wikidata.

Configuration changes

Wikidata configuration is in the operations/mediawiki-config repository.

The main files you need to edit are:

  • wmf-config/Wikibase.php: Wikidata specific configuration
  • wmf-config/InitialiseSettings.php: the options you want to enable for a specific wiki
  • wmf-config/CommonSettings.php: the logic matching IS options and their effect

You've basically two solutions to deploy your change:

  • Normally aude or hoo handles these
  • You can also sign up for a SWAT deploy where someone will deploy for you, point them to this page if they have Wikidata specific questions.

There are no deploys on Fridays (and weekends) except for emergencies. See below for the procedure to follow in this case, basically ping Greg or if missing RainbowSprinkles on Freenode #wikimedia-operations.

Schema changes

Schema changes should be reviewed by the WMF's database engineer. There are database specifics procedure and notes available at MediaWiki Database policy and at Schema changes.

In a nutshell, you need to know:

  • database schema change won't be applied as the same to the code
  • for any schema change, you need to create a task on Phabricator, tagged #DBA

After the schema change is done:

  • Add table to dumps: Add the table to the dump python scripts (https://phabricator.wikimedia.org/diffusion/ODUM/), by either poking Ariel (apergos) or making a patch yourself + poke Ariel. The 'ariel' branch of the dumps git repo is currently (as of July 2014) being used, so make the patch there. See 71087 for an example. A good description of the table is important. If the table contains any private information (e.g. ip addresses), then adding it to the dumps is more complicated and it would be best for Ariel to do it.
  • Add table to toollabs: The scripts for managing database replications on labs are in https://phabricator.wikimedia.org/diffusion/OSOF/. An example of how to add tables is https://gerrit.wikimedia.org/r/#/c/118582/ and how to have a column added is https://gerrit.wikimedia.org/r/#/c/125369/. Then poke DBAs and Labs team to review / deploy. If the table involves private information (e.g. ip addresses), then it is best just to poke Coren and Sean and let them handle it.

Update cron jobs

Dispatch changes and some other stuff is run as a cron job. Cron jobs are configured in puppet, in "manifests/misc/maintenance.pp".

https://phabricator.wikimedia.org/diffusion/OPUP/browse/master/manifests/misc/maintenance.pp;55e76106e0e976c5d56ccb2a54921ad90bbe764d

Any cron jobs that you want to stop need to be marked as "absent" and do not get removed from puppet manifest. That allows the servers to know this job must be removed where it exists.

To change a cron job:

  • if you need to edit the cron parameters, add a new cron job and mark the old one as "absent"
  • if you need to edit the script called by the cron job, prepare a change with this change, but the current cron job can be kept

How to make a Wikidata build for master / beta deployment

Builds for beta are generated daily on a labs instance, which is in the Wikidata-build project. Both WikidataJenkins and wmf-jenkins will run tests on any new builds. Once they both approve, then someone needs to +2 code review and then Jenkins will merge the new build. Builds for master branch of Wikidata.git will be deployed automatically to beta.

See also