Heterogeneous deployment

From Wikitech
Jump to: navigation, search
It has been suggested that this page should be merged with How to deploy code

Some of the information here is outdated. When in doubt, trust How to deploy code over this.

See also: "Heterogeneous deployment" page on mediawiki.org

See also: Heterogeneous deployment/old for information about older setup

This document describes the current deployment infrastructure, which allows for different wikis to be running different MediaWiki versions as the same time.

Where things live

WMF configuration lives in /srv/mediawiki-staging/wmf-config on the deployment host 'tin', and tools such as scap sync this directory to /usr/local/apache/common-local on cluster machines. (The same common directory also has subdirectories for each live branch of the MediaWiki codebase.) Most files are maintained in the git repository operations/mediawiki-config.


  • wikiversions.json: a JSON object mapping wikiname -> version. Versions must have a "php-" prefix, like "php-1.24wmf2". This is here for admins to edit as needed. Note that addwiki.php updates this automatically (rebuilding the cdb too).
  • wikiversions.php: a PHP version of the wikiversions.json file. Never edit this directly, it is not committed


Note: these files are all checked out from operations/mediawiki-config/multiversion
  • MWMultiVersion.php: Class for determining the database name and version of a wiki. Supports loading from three entry points:
    • Web access of an uploaded file through apache - setSiteInfoForUploadWiki()
    • Web access of a other wiki content through apache - setSiteInfoForWiki()
    • From within a maintenance script - setSiteInfoForMaintenance()
  • MWScript.php: Wrapper for running the right version of a maintenance script for a wiki.
  • updateWikiversions: switch all wikis in a .dblist to a given MediaWiki version in the common/wikiversions.json file.


This is where most of the site configuration lies. Main files include things like:

  • CommonSettings.php: Mostly handles shared settings or settings with few exceptions. Also handles the task of getting the InitialiseSettings.php options into globals that can be used for MW, and the caching thereof. Site, language, and DB name globals are extracted from the MWMultiVersion object. A MWMultiVersion object should always be initialized by the time this file is loaded. It also sets the i18n message directory locations for extensions. Other random things are performed here.
  • InitialiseSettings.php: Mostly contains MW configuration settings for individual wikis as well as wiki groups (like all "wikibooks" for example).
  • extension-list: The list of extensions available for use on the site (though some may not be enabled at the moment). These extensions must be installed on cluster. This list is used to automatically generate the ExtensionsMessages-*.php files needed for localization.
  • CommonSettings-labs.php, InitialiseSettings-labs.php, and other *-labs files contain overrides for the Beta Cluster (deployment) cluster.


This is where the various MW directories lie.


See Wikimedia binaries for a more exhaustive list.

  • scap: script to sync all hosts with the state of the deployment server (tin.eqiad.wmnet)

How to do stuff

General remarks on deploying code

Basic common sense

  • Be careful. Breaking the site is surprisingly easy!
  • If you're deploying code written by someone else, ask them to be around during deployment so they can troubleshoot if necessary.
  • Make sure you know about anything hairy, such as additional prerequisites (e.g. schema changes) or potential complications when rolling back.

Deployment Requirements

  • Cluster account request through Phabricator (set Security: Access Request!); request deployment access (requires manager and/or sr dev approval).
    • If you can ssh into tin, and ssh into a random mw box (e.g. mw1001) from there, you already have this.
  • Deployment branch merge access requested from an existing user in the group.
  • Some shiny new code with which to terrorize end-users.

Don't leave town

Even if your deploy appears to be working, it's important to be reachable in the hours immediately following your deploy. Ideally, stay online and in IRC channels like #wikimedia-tech for a couple of hours.

If you must go offline, let people know how to reach you (and keep your mobile phone or other communications device on your person). You can use /away messages on IRC, or perhaps send a short email to the ops list.

If you are on Wikimedia staff, now might be a great time to check if your contact info is up to date. If you aren't on staff, ask a staffer to add your contact info to that page, under "Important volunteers".

Basic code deployment steps

Really, you should check out the How to deploy code page. It is the canonical place for documentation on how to deploy code, obviously.

Change wiki configuration

See Configuration files for a list of configuration files and their formats

In your own repo via gerrit

If you need someone else to merge and push your change, follow this path.

  • for the person making the change
    • in your own checkout of the operations/mediawiki-config project, make the changes you need
    • push the change to gerrit for review
  • for the person merging and pushing the change live
    • on tin, cd to /srv/mediawiki-staging/
    • run git status and verify there are no pending changes. Untracked files are ok.
    • in gerrit: rebase, review, and merge the change
      • The rebase is to make sure the merge commit below isn't a false positive. Old patches will create FF-only merge commits.
    • on tin, run git fetch origin && git diff HEAD origin. Review the diff and make sure it's only the changes that you expect
      • if more than the changes you're making show up, STOP and find out why and make sure they're ok to merge.
    • if it looks good, run git merge origin/master
      • if this creates a merge commit, that means someone has committed a local change without pushing; this isn't the end of the world but it should be fixed, so either fix it yourself or ask for help
    • run scap sync-file wmf-config/filename 'Summary of change' for each changed file separately, scap sync-dir wmf-config 'Summary of change' if a lot of files in one directory changed, and/or scap sync-dir dblists/ 'Summary of change' if multiple dblist files changed

Change on tin then commit

If you are going to make the change, review, and push all by yourself, follow this path.

  • Edit the file(s) you need to edit.
  • cd to /srv/mediawiki-staging/
  • Run git diff wmf-config , which will display a diff of recent changes to the wmf-config directory. Verify that these changes are sane and that there are no traces of someone else messing with the same files
  • If everything looks good, run scap sync-file wmf-config/filename 'Summary of change' for each changed file separately, scap sync-dir wmf-config 'Summary of change' if a lot of files in one directory changed, and/or scap sync-dir dblists/ 'Summary of change' if multiple dblist files changed
  • Run git commit wmf-config/filename and add a decent commit summary. Then do git push origin HEAD:refs/for/master
  • Do not forget the git push step above, things will get messed up if you do (and nobody can review your changes in Gerrit)

Run a maintenance script on a wiki

To run a maintenance script, log into terbium, go into the directory for the version of MediaWiki that you want to run the maintenance script from (for example php-1.27.0-wmf.23):

cd /srv/mediawiki/php-1.27.0-wmf.23

Use the mwscript command-line utility:

mwscript scriptname.php dbname

The above command will run the script scriptname.php located in /maintenance since no directory was specified. You can also use a path relative to the MediaWiki installation, for example:

mwscript extensions/MyExtension/maintenance/scriptname.php dbname

Internally, this executes /home/wikipedia/common/multiversion/MWScript.php. It will run the version of the maintenance script in /home/wikipedia/common. If you want to run the version under /usr/local/apache/common-local/, or you're on a box without /home mounted, you need to change to /usr/local/apache/common-local/multiversion and run ./MWScript.php instead.


$ mwscript eval.php enwiki
> print $wgDBname;
> print $wgVersion;
> exit

If you need to pass parameters to your maintenance script, you can add them to the end. You may also want to explicitly declare the database parameter to avoid confusion:

mwscript scriptname.php --wiki=enwiki --days=1

In this case, the 'wiki' parameter is handled by mwscript and the 'days' parameter is handled by scriptname.php. The "wiki" must be in wikiversions.cdb so the maintenance script machinery can figure out what MediaWiki version it is running.

Many maintenance scripts are used to run heavy database queries; remember to check the replication lag and other possible byproducts.

Run a maintenance script on a group of wikis

If you want to run a maintenance script on all wikis in a .dblist file, you can issue the following command from anywhere:

mwscriptwikiset scriptname.php listname.dblist

This will run the script scriptname.php located in /maintenance since no directory was specified. As with MWScript.php and mwscript, you can also use a script path relative to the MediaWiki installation, for example:

mwscriptwikiset extensions/MyExtension/maintenance/scriptname.php listname.dblist

The .dblist file must be located in /usr/local/apache/common-local.

As with mwscript, mwscriptwikiset will run the "home" version of the maintenance script found in /home.

Switch a wiki to a different MW version

See /Train deploys for more modern instructions. TrustedXFF, dsh and scap-recompile are all dead tech.
  • First make sure that the new MW version exists. If not, see the section of this page concerned with checking out new versions.
  • First check if another wiki is already using the version. You can use the command mwversionsinuse --withdb. Note that this will only return *one* wiki using the version, not all of them.
  • If no other wiki is already using the MW version:
    • Locate the testwiki entry in wikiversions.json (on /a/) and change it to the new MW version. Make sure it has 'php-' in it.
    • Do cd /php-<MW VERSION>/extensions/TrustedXFF and then run php /srv/mediawiki-staging/multiversion/MWScript.php extensions/TrustedXFF/generate.php <WIKI NAME>
    • Run scap. This should update ExtensionMessages-*.php for the version.
    • Run dsh -F30 -cM -g mediawiki-installation -o -oSetupTimeout=10 'sudo -u mwdeploy /usr/bin/scap-recompile' to compile binaries for the version, like texvc.
    • Check that testwiki is operational.
  • Edit the wikiversions.json file by finding the wiki's DB name and changing the corresponding version value. Make sure it has 'php-' in it.
  • Run scap sync-wikiversions [DESCRIPTIVE MESSAGE] to rebuild the cdb from the json file and sync them.

Install a new extension on a wiki

If you're deploying an extension or feature that can be switched off, it's usually best to leave it switched off while you deploy and carefully switch it on after that using a simple configuration change (this is called a dark launch). Even if you do this, you should build any configuration infrastructure (e.g. $wmg variable, adding entry in InitialiseSettings with default false) at this time so all you'll have to do later is flip a switch.

For specific preparations, see the sections below as well as How to do a schema change and How to do a configuration change. Best to perform schema changes before making config changes.

Add a configuration switch for an extension

In /srv/mediawiki-staging/wmf-config/CommonSettings.php, add:

if ( $wmgEnableMyExtension ) {
	require_once( "$IP/extensions/MyExtension/MyExtension.php" );
	// Set config vars if needed

	// If you want to export config vars through InitialiseSettings.php, you need to set $wmgMyExtensionThingy there and do
	#$wgMyExtensionThingy = $wmgMyExtensionThingy;

In /srv/mediawiki-staging/wmf-config/InitialiseSettings.php, add something like:

'wmgEnableMyExtension' => array(
	'default' => false,
	'eswikibooks' => true,
	// etc.
// If needed, set $wmgMyExtensionWhatever vars here too

If your extension requires a large-ish amount of configuration, consider putting it in a separate file instead. Currently, AbuseFilter, LiquidThreads and FlaggedRevs do this.

For more documentation on these files and their formats, see Configuration files.

Add new extensions to extension-list

When adding a new extension, you need to add it to the extension list, or its i18n messages won't get picked up. For more information about this setup, see this.

  1. cd to /srv/mediawiki-staging/wmf-config
  2. Edit extension-list and add the path to the extension setup file on a line by itself
  3. Where X.XX is the current MediaWiki version, run mwscript maintenance/mergeMessageFileList.php --wiki=aawiki --list-file=extension-list --output=ExtensionMessages-X.XX.php and watch the output for errors. Your extension should appear in the output the same way the others do
    • If you get an error, you probably got the path wrong, or the extension setup file is wonky

Deploy the extension

Do what needs to be done to deploy and register changes in these config files. You will need to run scap afterwards (so the config and message cache is updated).

Add to make-wmf-branch

To have your extension added to newly created WMF branches, please add the newly added extension to make-wmf-branch/config.json in the release tools repository.

Make a wiki use a special version of an extension

This scenario occurs when we have MyExtension/ and MyExtensionBeta/ in one of the MW deployment directories, say php-X.XX, and we want some wikis using one extension version while other wikis use the other.

  • Edit wikiversions.json by finding the row with [target wikidb] [version] and changing it to [target wikidb] [version] [target wikidb]. This defines the wiki as having a separate i18n cache.
  • If this was already in wikiversions.json:
    • Edit CommonSettings.php to include the desired extension version if $wgDBname equals the target wiki and sync it
  • ...otherwise, perform the following steps:
    • Run refreshWikiversionsCDB to rebuild wikiversions.cdb.
    • cd to /srv/mediawiki-staging/wmf-config
    • Edit CommonSettings.php to include the desired extension version if $wgDBname equals the target wiki
    • Run mwscript mergeMessageFileList.php --wiki=TARGETWIKI --list-file=extension-list --output=ExtensionMessages-MWVERSIONFORTHATWIKI-TARGETWIKI.php and watch the output for errors. MWVERSIONFORTHATWIKI should not include "php-", it should be like "X.XX".
      • If you get an error, you probably got the path wrong, or the extension setup file is wonky.
      • If you get an error, you can also try running /home/catrope/sync-common , then running mergeMessageFileList.php again like above.
    • Check that a new ExtensionMessages-MWVERSIONFORTHATWIKI-TARGETWIKI.php file exists in wmf-config/ and makes sense. If it does, then svn commit the file.
    • Run scap.

Switch all wikis of one version to a different MW version

  • First make sure that the new MW version exists. If not, see the section of this page concerned with checking out new versions.
  • If no other wiki is already using the MW version then you probably should stop what you're doing.
  • Run /srv/mediawiki-staging/multiversion/updateWikiversions all php-NEWVERSION to switch all the wikis to using a new version. This can be for upgrades or for downgrades. This will rebuild the wikiversions.json and wikiversions.cdb files.
  • Run scap.

Rebuild the wikiversions.cdb file

Run refreshWikiversionsCDB to rebuild the local cdb from the json file. More commonly you will want to run scap sync-wikiversions to compile and sync wikiversions.cdb to all hosts in the cluster.

Create a new MediaWiki version

See the Train deploys subpage for full details.