Wikimedia binaries

From Wikitech
Jump to: navigation, search

The following executable binaries are maintained in version control (operations/puppet:/files/misc/scripts) and copied/synced to servers (such as tin) by puppet into /usr/local/bin.

bin

apache-fast-test

This script comes from puppet.

fatalmonitor

  • Runs a watch command on the hhvm logs with various exclusion and aggregation filters to monitor current php notices, warnings and errors on the cluster. This script lives on mwlog1001.

scap pull

Note: this script is from mediawiki/tools/scap.

Performs the action only on the one server from which the command is ran
  • syncs /srv/mediawiki-staging from a deployment rsync server-> /srv/mediawiki on the local server
  • This action was formerly run as sync-common

scap sync-dir

Note: this script is from mediawiki/tools/scap.

This is an all-script!
  • This command is deprecated and will be removed in a future scap release. Use scap sync-file instead.
  • for a single directory
  • checks PHP syntax
  • syncs /srv/mediawiki-staging/(some dir) -> /srv/mediawiki
  • Runs the same code internally as scap sync-file

scap sync-file

Note: this script is from mediawiki/tools/scap.

This is an all-script!
  • for a single file
  • checks PHP syntax
  • syncs /srv/mediawiki-staging/(some file) -> /srv/mediawiki

scap sync-wikiversions

Note: this script is from mediawiki/tools/scap.

This is an all-script!
  • syncs /srv/mediawiki-staging/wikiversions.{json,cdb} -> /srv/mediawiki

scap sync

shortcut: scap
'scap' logo
A diagram of wikimedia 'scap' deployment tool

Note: this script is from mediawiki/tools/scap. Generated documentation is published on doc.wikimedia.org

This is an all-script!
"sync-common-all-php"
  • lints PHP files in ./wmf-config, ./multiversion
  • Sync deploy directory (/srv/mediawiki-staging) on localhost with staging area (/srv/mediawiki)
  • for all mediawiki versions currently deployed (usually 2) it:
    • rebuilds localization caches of core and extensions
    • creates json files from localization caches for better rsync performance
  • runs scap pull on co-master servers to update /srv/mediawiki-staging
  • runs scap pull on rsync fanout servers to update /srv/mediawiki
  • runs scap pull on "all" servers to update /srv/mediawiki from nearest rsync slave
  • rebuilds localization caches on "all" servers from json files that were synced by scap pull
  • compiles wikiversions.json to wikiversions.cdb on localhost
  • runs scap sync-wikiversions
bd808@silver:~$ scap say
 ------------------------------------------------
/                                                \
|     S.C.A.P.: someone can always pontificate    |
\                                                /
 ------------------------------------------------
    \
     \
      \
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

mwgrep

shortcut: mwgrep

Search for CSS and JavaScript code fragments in MediaWiki, User and Module namespaces pages on all Wikimedia projects (powered by Elasticsearch).

It's located in operations/puppet/modules/scap/files/mwgrep

expanddblist

For convenience, this utility is also available as bin/expanddblist in any checkout of operations/mediawiki-config.git.
shortcut: mwgrep
Usage: expanddblist [<dblist>, <dblistname> or <expression>]

Example:

./bin/expanddblist flow_computed > dblists/flow.dblist

All-script

An all-script is a script that operates on mediawiki-installation boxes via ssh. These scripts performs the action on all servers.

The specific servers include:

Purge

Sometimes you may need to purge a URL from Varnish cache.

In most cases this is enough. However if the change involves more than a simple change to redirects, depooling/ repooling may be necessary. Make sure you find out whether this is needed ahead of time (while at it, start the depooling page and link to it, we need documentation!).

See also