Nova Resource:Deployment-prep/Overview
- Reviewed in September 2016
The Beta Cluster project reproduces a good part of the WMF production cluster so we can use it as a test and staging area for code changes and new extensions. The Beta Cluster is hosted in the Wikimedia Cloud VPS virtual environment where new virtual machines are automatically configured by Puppet using the Beta Cluster's puppetmaster.
The MediaWiki installation in Beta Cluster are automatically kept up-to-date with the master branches of MediaWiki core and extensions.
Architecture
DNS
There is a domain called beta.wmflabs.org. DNS and public IP allocations are entirely managed through Horizon.
- it contains a wildcard (*) record pointing to
deployment-cache-text05. upload.wikimedia.beta.wmflabs.orgpoints todeployment-cache-upload05.irc.beta.wmflabs.orgpoints todeployment-ircd.
Varnish
The "text" Varnish traffic (*.beta.wmflabs.org) is served by the deployment-cache-text05 instance which load balances requests to one of two backend Apaches. Apaches receive their MediaWiki installation and configuration using Scap from deployment-deploy01.
The load balancing is done by having Apaches listed as Varnish backends in the puppet cache configuration.
Apache
The Apaches serve MediaWiki requests.
The configuration is available in operations/puppet.git, in the branch production, folder modules/mediawiki/files/apache/beta/sites/.
Unlike in production, Apache access log are actually enabled and written locally to /var/log/apache2/other_vhosts_access.log on each host.
MediaWiki
Wikis follow the same <language>.<project>.beta.wmflabs.org hostname schema as production wikis. The Meta-Wiki site of Beta Cluster is thus served via https://meta.wikimedia.beta.wmflabs.org/.
The mobile version uses <language>.m.<project>.beta.wmflabs.org, matching production. The mobile version of enwiki is thus at https://en.m.wikipedia.beta.wmflabs.org/.
The Beta Cluster runs the master branch of MediaWiki and extensions. The code is updated automatically by Jenkins jobs. See How code is updated for more details.
The database hosts for MediaWiki are deployment-db11 to deployment-db14.
The Memcached hosts for MediaWiki are deployment-memc08 to deployment-memc10.
The Redis host for MediaWiki is deployment-rdb-01.
Swift
Media files and thumbnails for MediaWiki are served by an "upload" Varnish (deployment-cache-upload05). This uses deployment-ms-fe02 as a backend, which connects to deployment-cache-text05 in case it can't find a requested file in Swift.
IRC
irc.beta.wmflabs.org runs ircd, accepting messages from labs MediaWiki servers. For example, edits made to https://en.wikipedia.beta.wmflabs.org/ are broadcasted to the #en.wikipedia.beta.wmflabs channel and so on.
Logs
Various server logs are written to the remote syslog server deployment-mwlog01 in /srv/mw-log. This matches the production cluster setup (except the production log host, mwlog1001, has mw-log under /a instead of /srv). See the Logs page for a description of some of the logs we have in production, most of which should exist in the Beta Cluster as well.
Configure
See also How code is updated
The work machine is deployment-deploy03, basically an equivalent to production deployment servers. (To access Cloud VPS instances, see Help:Access.)
File structure
on the deployment server: /srv/mediawiki-staging => operations/mediawiki-config.git on the apaches: /srv/mediawiki => operations/mediawiki-config.git on the puppetmaster: /var/lib/git/operations/puppet => operations/puppet.git on the puppetmaster: /var/lib/git/labs/private => labs/private.git /srv/mediawiki-staging/php-master => mediawiki/core.git /srv/mediawiki-staging/php-master/extensions => mediawiki/extensions.git
Thus to view beta cluster logs such as exception.log and fatal.log, log-in to the server and navigate /srv/mw-log:
home$ ssh deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud deployment-deploy03$ cd /srv/mw-log
The git repositories are maintained automatically by Jenkins jobs, see how code is updated.
MediaWiki config
The MediaWiki configuration files are located on /srv/mediawiki-staging. It is a clone of the operations/mediawiki-config.git repository and is maintained automatically by Jenkins (see how code is updated). Any local hacks/changes will get reverted when Jenkins updates the code, so if you do any hack, make sure to send it in Gerrit or your change will be lost.
You will mostly want to add your configuration changes in wmf-config/InitialiseSettings-labs.php or wmf-config/CommonSettings-labs.php and it will need to be reviewed via Gerrit by a production deployer/ops. Once merged the change will be applied automatically, but will still need to be merged on production to prevent alerts from being triggered.
There is a single local hack which is the private directory. It holds secret keys and passwords. git happily ignores the file and it must not be made public.
Database connection
To connect to a database use the sql wrapper:
sql testwiki sql wikishared
To get a write connection (e.g. create a new table on wikishared which canât happen through update.php):
sql wikishared --write
New extension
Extensions live in /srv/mediawiki-staging/php-master/extensions, which is a clone of mediawiki/extensions.git. Which means that to get an extension deployed on the beta cluster it must be registered as a submodule of the mediawiki/extensions.git repository. It will then be automatically applied.
You will need to do all the steps needed for a production deployment:
- add a configuration snippet in
operations/mediawiki-config.git:wmf-confg/CommonSettings-labs.php - add the entry point for the message cache to load the i18n files. It would be added to
wmf-config/extension-listorextension-list-labs(if it is labs only)
Once the change is merged, the Jenkins jobs will take care of refreshing the l10n cache and invalidating the serialized configuration by touching InitiaseSettings.php.
You might have to apply schema changes, for now with mwscript update.php $SOMEWIKIDB. The foreachwiki script might work too. A Jenkins job also does this hourly (see how code is updated).