Nova Resource:Deployment-prep/Overview

From Wikitech
Jump to: navigation, search
Reviewed in September 2016

The mw:Beta Cluster project reproduces a good part of the WMF production cluster so we can use it as a test and staging area for code changes and new extensions. It runs on the wmflabs.org virtual cluster which provides us with a way to easily create virtual machines that are automatically configured by puppet using our puppetmaster. The MediaWiki and MediaWiki extensions are kept up to date to the tip of their master branches.

Architecture

There is a domain called beta.wmflabs.org which contains a '*' record pointing to deployment-cache-text04 and an 'upload' record pointing to deployment-cache-upload04. DNS and public IP allocations are entirely managed through Horizon.

Wikis follow the <language>.<project>.beta.wmflabs.org hostname schema. The beta Meta Wikimedia is thus served via https://meta.wikimedia.beta.wmflabs.org/.

The mobile version uses <language>.m.<project>.beta.wmflabs.org like in production. The mobile version of enwiki is thus at https://en.m.wikipedia.beta.wmflabs.org/.

The text traffic (*.beta.wmflabs.org) is served by the deployment-cache-text04 instance which load balance requests to two backend Apaches. A third backend Apache exists for automated security testing by clients sending an X-Wikimedia-Security-Audit: 1 HTTP header. Apaches receive their configuration and MediaWiki install from scap on deployment-tin. This includes both normal MediaWiki and mobile (*.m.*.beta.wmflabs.org) traffic.

MediaWiki and its extensions are running out of their master branches and are updated automatically by Jenkins jobs. See How code is updated for more details.

Data can be cached in the memcached instances (deployment-memc04 and deployment-memc05) which run on dedicated instances or are fetched from the MariaDB database hosted on deployment-db1 or deployment-db2.

Files and thumbnails are served by a Varnish upload cache (deployment-cache-upload04). This uses deployment-ms-fe01 as a backend, which connects to deployment-cache-text04 in case it can't find a requested file in Swift.

irc.beta.wmflabs.org points to deployment-ircd, runs ircd and udpmxircecho, accepting accepts data from labs, broadcasting them on the ircd instance. Edits on http://deployment.wikimedia.beta.wmflabs.org/ go to the #deployment.wikimedia.beta.wmflabs channel and so on.

Components

This section describe the components for each layer starting from page caches till the database.

Varnish (text cache)

The text cache is served by Varnish on the deployment-cache-text04 instance.

The load balancing is done by having Apaches listed as Varnish backends in the puppet cache configuration.

Apaches

The Apaches serve MediaWiki requests.

The configuration is available in operations/puppet.git, in the branch production, folder modules/mediawiki/files/apache/beta/sites/.

Logs

Various server logs are written to the remote syslog server deployment-fluorine02 in /srv/mw-log. This matches the production cluster setup (except the production log host, mwlog1001, has mw-log under /a instead of /srv). Logs describes some of the production logs, and many have equivalents on the beta cluster.

Apache access logs are written to /var/log/apache2/other_vhosts_access.log on each host.

Configure

See also how code is updated

The work machine is deployment-tin, basically an equivalent to tin in production. (To access machines on our internal wmflabs network, see Help:Access.)

File structure

on the deployment server: /srv/mediawiki-staging => operations/mediawiki-config.git
on the apaches: /srv/mediawiki => operations/mediawiki-config.git
on the puppetmaster: /var/lib/git/operations/puppet => operations/puppet.git
on the puppetmaster: /var/lib/git/labs/private => labs/private.git
/srv/mediawiki-staging/php-master => mediawiki/core.git
/srv/mediawiki-staging/php-master/extensions => mediawiki/extensions.git

Thus to view beta cluster logs such as exception.log and fatal.log, ssh deployment-fluorine02.deployment-prep.eqiad.wmflabs and view /srv/mw-log.

The git repositories are maintained automatically by Jenkins jobs, see how code is updated.

MediaWiki config

The MediaWiki configuration files are located on /srv/mediawiki-staging. It is a clone of the operations/mediawiki-config.git repository and is maintained automatically by Jenkins (see how code is updated). Any local hacks/changes will get reverted when Jenkins updates the code, so if you do any hack, make sure to send it in Gerrit or your change will be lost.

You will mostly want to add your configuration changes in wmf-config/InitialiseSettings-labs.php or wmf-config/CommonSettings-labs.php and it will need to be reviewed via Gerrit by a production deployer/ops. Once merged the change will be applied automatically, but will still need to be merged on production tin to prevent alerts from being triggered.

There is a single local hack which is the private directory. It holds secret keys and passwords. git happily ignores the file and it must not be made public.

Database connection

To connect to a database use the mwscript wrapper and MediaWiki sql.php script:

mwscript sql.php deploymentwiki

New extension

Extensions live in /srv/mediawiki-staging/php-master/extensions, which is a clone of mediawiki/extensions.git. Which means that to get an extension deployed on the beta cluster it must be registered as a submodule of the mediawiki/extensions.git repository. It will then be automatically applied.

You will need to do all the steps needed for a production deployment:

  1. add a configuration snippet in operations/mediawiki-config.git:wmf-confg/CommonSettings-labs.php
  2. add the entry point for the message cache to load the i18n files. It would be added to wmf-config/extension-list or extension-list-labs (if it is labs only)

Once the change is merged, the Jenkins jobs will take care of refreshing the l10n cache and invalidating the serialized configuration by touching InitiaseSettings.php.

You might have to apply schema changes, for now with mwscript update.php $SOMEWIKIDB. The foreachwiki script might work too. A Jenkins job also does this hourly (see how code is updated).

New wiki

See Nova Resource:Deployment-prep/Add_a_wiki.