Deployments/Training

Deployment training paused, pending a retrospective (see: task T354318)

Find deployment training on the Google Deployment Calendar wikimedia.org_rudis09ii2mm5fk4hgdjeh1u64@group.calendar.google.com

Welcome, potential deployer!

We're happy you're here! Whether you're new to deploying, or just need a refresher on the finer points of the process for Wikimedia's production infrastructure: you've come to the right place. Our goal is to make deployments low-risk and a big component of that is incremental releases via backport.

The deployment system is never done: Write the doc you wish you had when you started, record a screencast, file a task for scap, find a relenger to rant at, make a patch — all encouraged!

But first, get training!

Get training

There are two Backport/config deployment training windows per week. During these windows volunteers can help walk through the steps of deployment of any of the patches on the schedule.

Look for UTC early backport and config training or UTC evening backport and config training in the Deployments schedule. There is also a Google meet scheduled during these windows, so that you can talk with the trainers live.

Sign up using the deployment training request form and be ready to take notes, to ssh into the deployment and mwdebug hosts, and to run commands.

You'll get to walk through one or more deployments of a backported MediaWiki change or a configuration change, starting with merging it, fetching it onto the deployment host, checking that it merges in cleanly and there's no uncommitted stuff around, putting it out to an mwdebug host for testing, and finally scapping it out to the entire cluster, all under the supervision of experienced deployers.

Don't be shy about asking questions. Have a question about how to manage a config change that includes multiple files? This is your opportunity to ask all about it, take notes, and help improve the docs about deployment!

If one training isn't enough, keep coming back. Do more deployments with someone more experienced around in case you run into "OMG how do I revert this?" and other such issues. That is what these training windows are for. Deploy, deploy, deploy!

When you've been through the training and you feel comfortable enough with deployments, you can help others by becoming a trainer. There's no official process to this; just sign up by adding your name as a deployer during the training window that's in the most convenient timezone, and don't forget to show up. Future deployers thank you in advance :-)

A summary of useful commands and other information is given below, and there is more detailed documentation available at Backport_windows/Deployers in case you prefer to read up in detail about everything ahead of time.

Useful information

Usual method using scap backport

• scap backport <change_number_or_url>

Basic sync

• All git activity happens in /srv/mediawiki-staging
• +2 this patch
• git fetch
• git log -p HEAD..@{u}
• scap sync-file [FILE] '[Message]'

Mw/Config change More advanced

Most of the time, you'll want to check the patch before it goes out. We do this on mwdebug servers.

• +2 patch
• git fetch
• git log -p HEAD..@{u}
• git rebase
• [in another terminal] ssh mwdebug1002.eqiad.wmnet && scap pull
• [check in browser]
• scap sync-file [FILE] '[Message]'

MediaWiki patch

• +2 patch for a deployed version of MediaWiki, skins, or extensions
• cd /srv/mediawiki-staging/php-[version]/ OR cd /srv/mediawiki-staging/php-[version]/extension
• git fetch
• git log -p HEAD..@{u}
• git rebase
• [if extension] git submodule update
• [in another window] ssh mwdebug1002.eqiad.wmnet && scap pull
• [check in browser]
• scap sync-file [FILE] '[Message]'

Helpful things to know

Deployment server directory structure

• /srv/mediawiki-staging is a git clone of operations/mediawiki-config
• /srv/mediawiki-staging/php-[version] is a separate git clone of mediawiki/core not a submodule
  • /srv/mediawiki-staging/php-[version]/extensions/[extension] is a submodule of php-[version]
  • /srv/mediawiki-staging/php-[version]/skins/[skin] is a submodule of php-[version]

/srv/mediawiki-staging
├── .git
├── dblists
│   ├── all.dblist
|   └── [...]
├── php -> php-1.36.0-wmf.26
├── php-1.36.0-wmf.36 # This is a MediaWiki install
│   ├── .git
│   ├── [...]
│   ├── extensions
│   │   ├── 3D/.git # git submodule to mediawiki/extensions/3D.git
│   │   └── XAnalytics/.git
│   │   ├── [...] 100s of extensions
│   ├── skins
│   │   ├── CologneBlue/.git
│   │   └── Vector/.git
│   │   ├── [...] 8 skins
│   ├── includes
│   │   └── [...]
│   ├── index.php
│   ├── api.php
|   └── load.php
├── portals
├── private
|   ├── [...]
│   └── PrivateSettings.php
├── README
├── wikiversions.json
└── wmf-config
    ├── CommonSettings.php
    ├── InitialiseSettings.php
    └── [...]