Nova Resource:Deployment-prep/Overview

Reviewed in September 2016

The Beta Cluster project reproduces a good part of the WMF production cluster so we can use it as a test and staging area for code changes and new extensions. The Beta Cluster is hosted in the Wikimedia Cloud VPS virtual environment where new virtual machines are automatically configured by Puppet using the Beta Cluster's puppetserver.

The MediaWiki installation in Beta Cluster are automatically kept up-to-date with the master branches of MediaWiki core and extensions.

Architecture

DNS

There is a domain called beta.wmflabs.org. DNS and public IP allocations are entirely managed through Horizon.

it contains a wildcard (*) record pointing to deployment-cache-text05 .
upload.wikimedia.beta.wmflabs.org points to deployment-cache-upload05.
irc.beta.wmflabs.org points to deployment-ircd.

Varnish

Apache

The Apaches serve MediaWiki requests.

The configuration is available in operations/puppet.git, in the branch production, folder modules/mediawiki/files/apache/beta/sites/.

Unlike in production, Apache access log are actually enabled and written locally to /var/log/apache2/other_vhosts_access.log on each host.

MediaWiki

Wikis follow the same <language>.<project>.beta.wmflabs.org hostname schema as production wikis. The Meta-Wiki site of Beta Cluster is thus served via https://meta.wikimedia.beta.wmflabs.org/.

The mobile version uses <language>.m.<project>.beta.wmflabs.org, matching production. The mobile version of enwiki is thus at https://en.m.wikipedia.beta.wmflabs.org/.

The Beta Cluster runs the master branch of MediaWiki and extensions. The code is updated automatically by Jenkins jobs. See How code is updated for more details.

The database hosts for MediaWiki are deployment-db11 to deployment-db14.

The Memcached hosts for MediaWiki are deployment-memc08 to deployment-memc10.

The Redis host for MediaWiki is deployment-rdb-01.

Swift

IRC

Logs

Various server logs are written to the remote syslog server deployment-mwlog01 in /srv/mw-log. This matches the production cluster setup (except the production log host, mwlog1001, has mw-log under /a instead of /srv). See the Logs page for a description of some of the logs we have in production, most of which should exist in the Beta Cluster as well.

Configure

See also How code is updated

The work machine is deployment-deploy04, basically an equivalent to production deployment servers. (To access Cloud VPS instances, see Help:Access.)

File structure

on the deployment server: /srv/mediawiki-staging => operations/mediawiki-config.git
on the apaches: /srv/mediawiki => operations/mediawiki-config.git
on the puppetserver: /srv/git/operations/puppet => operations/puppet.git
on the puppetserver: /srv/git/labs/private => labs/private.git
/srv/mediawiki-staging/php-master => mediawiki/core.git
/srv/mediawiki-staging/php-master/extensions => mediawiki/extensions.git

Thus to view beta cluster logs such as exception.log and fatal.log, log-in to the server and navigate /srv/mw-log:

home$ ssh deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud
deployment-deploy04$ cd /srv/mw-log

The git repositories are maintained automatically by Jenkins jobs, see how code is updated.

MediaWiki config

The MediaWiki configuration files are located on /srv/mediawiki-staging. It is a clone of the operations/mediawiki-config.git repository and is maintained automatically by Jenkins (see how code is updated). Any local hacks/changes will get reverted when Jenkins updates the code, so if you do any hack, make sure to send it in Gerrit or your change will be lost.

You will mostly want to add your configuration changes in wmf-config/InitialiseSettings-labs.php or wmf-config/CommonSettings-labs.php and it will need to be reviewed via Gerrit by a production deployer/ops. Once merged the change will be applied automatically, but will still need to be merged on production to prevent alerts from being triggered.

Like in production, the private directory is its own git repository. It holds secret keys and passwords, mainly in the PrivateSettings.php file. The contents of this directory must not be made public.

Database connection

To connect to a database use the sql wrapper:

sql testwiki
sql wikishared

To get a write connection (e.g. create a new table on wikishared which can’t happen through update.php):

sql wikishared --write

New extension

Extensions live in /srv/mediawiki-staging/php-master/extensions, which is a clone of mediawiki/extensions.git. Which means that to get an extension deployed on the beta cluster it must be registered as a submodule of the mediawiki/extensions.git repository. It will then be automatically applied.

You will need to do all the steps needed for a production deployment:

add a configuration snippet in operations/mediawiki-config.git:wmf-confg/CommonSettings-labs.php
add the entry point for the message cache to load the i18n files. It would be added to wmf-config/extension-list or extension-list-labs (if it is labs only)

Once the change is merged, the Jenkins jobs will take care of refreshing the l10n cache and invalidating the serialized configuration by touching InitiaseSettings.php.

You might have to apply schema changes, for now with mwscript update.php $SOMEWIKIDB. The foreachwiki script might work too. A Jenkins job also does this hourly (see how code is updated).

New wiki

See Nova Resource:Deployment-prep/Add_a_wiki.