Obsolete:LocalisationUpdate

This page contains historical information. It may be outdated or unreliable.

2018

Purpose

This MediaWiki extension fetches updates to MediaWiki core and extension localization files and makes them available to an installation without updating its own source. This allows us to pull updated translations being committed to development trunk even though our live install is running a deployment branch.

Source

• http://www.mediawiki.org/wiki/Extension:LocalisationUpdate

Big picture overview

• Translators tear it up on translatewiki.net
• translatewiki.net staff commit translations to master branch of core and extensions
• LocalisationUpdate cron job runs Monday–Thursday nights at 02:00 UTC on deploy1001.eqiad.wmnet
  • Pulls translations from local git clones and writes updated messages to a cache directory
  • Rebuilds the l10n cache(s) including updates found above
• Syncs the l10n cache(s) to apaches

What the cron job on tin does

• Runs as the l10nupdate user. This user has a passphraseless SSH key
• Updates the git clones in /var/lib/l10nupdate/mediawiki/core and /var/lib/l10nupdate/mediawiki/extensions
• Runs /srv/mediawiki-staging/php-$VERSION/extensions/LocalisationUpdate/update.php for each MediaWiki version
  • For core and for each extension, this script reads the i18n file(s) from the trunk checkout and compares them with the current translations on the cluster
  • If a message has a new/updated translation, and the English version of the message has not changed, updates that message
  • Writes message updates for each language to /srv/mediawiki-staging/php-$VERSION/cache/l10n/l10nupdate-$LANGCODE.json
• Runs /usr/local/bin/sync-l10nupdate which syncs /srv/mediawiki-staging/php-$VERSION/cache/l10n on tin to /srv/mediawiki/php-$VERSION/cache/l10n on the Apaches.
• Runs /srv/mediawiki-staging/php-$VERSION/maintenance/wmf/clearMessageBlobs.php, which clears message blobs cached by ResourceLoader
• Logs to /var/log/l10nupdatelog/l10nupdate.log in append mode. Log file is rotated by logrotate.
• logmsgbot !log's success or failure in #wikimedia-operations and Server Admin Log via stashbot (search)

Running LU manually

Run l10nupdate on deploy1001.eqiad.wmnet. Syncing the message updates requires deploy access (i.e. ssh access to the Apaches).

For the WikimediaMessages extension, if you need to add a NEW message, you previously need to backport the change adding this message to the relevant wmf branch.

Files

• /usr/local/bin/l10nupdate-1
  • Script that does all the work. The cron job runs this as the l10nupdate user
• /usr/local/bin/l10nupdate
  • Wrapper that runs l10nupdate-1 as the l10nupdate user using sudo
• /var/lib/l10nupdate/mediawiki
  • MediaWiki checkouts of trunk core and extensions
• /srv/mediawiki-staging/php-$VERSION/cache/l10n
  • Storage for serialized updates
• /srv/mediawiki/php-$VERSION/cache/l10n
  • Synced home of serialized updates per web server
• /usr/local/bin/sync-l10nupdate-1
  • Script that syncs the updates to the Apaches
• /usr/local/bin/sync-l10nupdate
  • Wrapper that runs sync-l10nupdate-1 as the l10nupdate user using sudo

Where this are in puppet

https://github.com/wikimedia/puppet/blob/production/modules/scap/manifests/l10nupdate.pp

• misc::l10nupdate (in manifests/misc-servers.pp)
  • Installs cron job
  • Installs l10nupdate, l10nupdate-1, sync-l10nupdate and sync-l10nupdate-1 scripts
  • Creates logging directory and SVN checkout directory
  • Installs logrotate config file
• files/misc/l10nupdate
  • Scripts live here
• files/logrotate/l10nupdate
  • logrotate config file

Config

MediaWiki configuration variables:

• $wmgUseLocalisationUpdate (to enable/disable extension; lives in InitialiseSettings.php)
• $wgLocalisationUpdateDirectory = "/var/lib/l10nupdate/caches/cache-$wmgVersionNumber"; (set in CommonSettings.php)