Nova Resource:Deployment-prep/Overview

Last reviewed in September 2016

The Beta Cluster is a staging environment for MediaWiki wikis hosted in production by Wikimedia Foundation, allowing the testing and development of software changes, infrastructure changes, configuration changes, and on-site changes (e.g. templates, gadgets). It reproduces the MediaWiki stack (Apache, PHP, MySQL) and good chunk of supporting infrastructure around it such as CDN (Varnish), Multimedia (Swift, Thumbor), and more. The Beta Cluster is hosted in the Wikimedia Cloud VPS virtual environment where new virtual machines are automatically configured by Puppet using the Beta Cluster's puppetserver.

Commits to the Git master branches of MediaWiki core and extensions repositories are automatically deployed to the Beta Cluster within 5 minutes.

Services

Service	Production URL	Beta Cluster equivalent
MediaWiki	https://en.wikipedia.org https://commons.wikimedia.org https://www.wikidata.org	https://en.wikipedia.beta.wmcloud.org https://commons.wikimedia.beta.wmcloud.org https://www.wikidata.beta.wmcloud.org
UrlShortener	https://w.wiki https://w.wiki/4	https://w.beta.wmcloud.org https://w.beta.wmcloud.org/4
Swift	https://upload.wikimedia.org	https://upload.wikimedia.beta.wmcloud.org
EventStreams	https://stream.wikimedia.org/v2/ui/ https://stream.wikimedia.org/?doc#/streams	https://stream.wikimedia.beta.wmcloud.org/v2/ui/ https://stream.wikimedia.beta.wmcloud.org/?doc#/streams

Architecture

DNS

There is a domain called beta.wmcloud.org. DNS and public IP allocations are entirely managed through Horizon.

• it contains a wildcard (*) record pointing to deployment-cache-text05 .
• upload.wikimedia.beta.wmcloud.org points to deployment-cache-upload05.
• irc.wikimedia.beta.wmcloud.org points to deployment-ircd.

Varnish

See also: Varnish

The "text" Varnish traffic (*.beta.wmcloud.org) is served by the deployment-cache-text05 instance which load balances requests to one of two backend Apaches. Apaches receive their MediaWiki installation and configuration using Scap from deployment-deploy01.

The load balancing is done by having Apaches listed as Varnish backends in the puppet cache configuration.

Apache

The Apaches serve MediaWiki requests.

The configuration is available in operations/puppet.git, in the branch production, folder modules/mediawiki/files/apache/beta/sites/.

Unlike in production, Apache access log are actually enabled and written locally to /var/log/apache2/other_vhosts_access.log on each host.

MediaWiki

Wikis follow the same <language>.<project>.beta.wmcloud.org hostname schema as production wikis. The Meta-Wiki site of Beta Cluster is thus served via https://meta.wikimedia.beta.wmcloud.org/.

The mobile version uses <language>.m.<project>.beta.wmcloud.org, matching production. The mobile version of enwiki is thus at https://en.m.wikipedia.beta.wmcloud.org/.

The Beta Cluster runs the master branch of MediaWiki and extensions. The code is updated automatically by Jenkins jobs. See How code is updated for more details.

The database hosts for MediaWiki are deployment-db11 to deployment-db14.

The Memcached hosts for MediaWiki are deployment-memc08 to deployment-memc10.

The Redis host for MediaWiki is deployment-rdb-01.

Swift

See also: Swift

Media files and thumbnails for MediaWiki are served by an "upload" Varnish (deployment-cache-upload05). This uses deployment-ms-fe02 as a backend, which connects to deployment-cache-text05 in case it can't find a requested file in Swift.

IRC

See also: irc.wikimedia.org

irc.wikimedia.beta.wmcloud.org runs ircd, accepting messages from labs MediaWiki servers. For example, edits made to https://en.wikipedia.beta.wmcloud.org/ are broadcasted to the #en.wikipedia channel and so on.

Logs

Various server logs are written to the remote syslog server deployment-mwlog01 in /srv/mw-log. This matches the production cluster setup (except the production log host, mwlog1001, has mw-log under /a instead of /srv). See the Logs page for a description of some of the logs we have in production, most of which should exist in the Beta Cluster as well.

Configure

See also How code is updated

The work machine is deployment-deploy04, basically an equivalent to production deployment servers. To access Cloud VPS instances, see Help:Access.

Puppet

See also: Nova Resource:Deployment-prep/How code is updated#Cherry-picking a patch from gerrit

File structure

on the deployment server: /srv/mediawiki-staging => operations/mediawiki-config.git
on the apaches: /srv/mediawiki => operations/mediawiki-config.git
on the puppetserver: /srv/git/operations/puppet => operations/puppet.git
on the puppetserver: /srv/git/labs/private => labs/private.git
/srv/mediawiki-staging/php-master => mediawiki/core.git
/srv/mediawiki-staging/php-master/extensions => mediawiki/extensions.git

Thus to view beta cluster logs such as exception.log and fatal.log, log-in to the server and navigate /srv/mw-log:

home$ ssh deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud
deployment-deploy04$ cd /srv/mw-log

The git repositories are maintained automatically by Jenkins jobs, see how code is updated.

MediaWiki config

The MediaWiki configuration files are located on /srv/mediawiki-staging. It is a clone of the operations/mediawiki-config.git repository and is maintained automatically by Jenkins (see how code is updated). Any local hacks/changes will get reverted when Jenkins updates the code, so if you do any hack, make sure to send it in Gerrit or your change will be lost.

You will mostly want to add your configuration changes in wmf-config/InitialiseSettings-labs.php or wmf-config/CommonSettings-labs.php and it will need to be reviewed via Gerrit by a production deployer/ops. Once merged the change will be applied automatically, but will still need to be merged on production to prevent alerts from being triggered.

Like in production, the private directory is its own git repository. It holds secret keys and passwords, mainly in the PrivateSettings.php file. The contents of this directory must not be made public.

Database connection

To connect to a database use the sql wrapper:

sql testwiki
sql wikishared

To get a write connection (e.g. create a new table on wikishared which can’t happen through update.php):

sql wikishared --write

New extension

Extensions live in /srv/mediawiki-staging/php-master/extensions, which is a clone of mediawiki/extensions.git. Which means that to get an extension deployed on the beta cluster it must be registered as a submodule of the mediawiki/extensions.git repository. It will then be automatically applied.

You will need to do all the steps needed for a production deployment:

1. add a configuration snippet in operations/mediawiki-config.git:wmf-confg/CommonSettings-labs.php
2. add the entry point for the message cache to load the i18n files. It would be added to wmf-config/extension-list or extension-list-labs (if it is labs only)

Once the change is merged, the Jenkins jobs will take care of refreshing the l10n cache and invalidating the serialized configuration by touching InitiaseSettings.php.

You might have to apply schema changes, for now with mwscript update.php $SOMEWIKIDB. The foreachwiki script might work too. A Jenkins job also does this hourly (see how code is updated).

New wiki

See Nova Resource:Deployment-prep/Add_a_wiki.

See also

• Beta Cluster on mediawiki.org