SWAT deploys/Deployers

From Wikitech
Jump to navigation Jump to search

This document is intended to provide detailed instructions for SWAT deployers. Hopefully, this document will prove useful to new SWAT deployers as well as provide a place for more experienced deployers to take notes on any tips and tricks they have discovered in the course of doing deployments.

General advice

  • Claim the SWAT window early to avoid confusion.
When jouncebot pings deployers in #wikimedia-operations connect, if you want to run that SWAT, say so I can SWAT today!
  • Try to think out loud and be explicit.
If you are nervous about deploying a particular patch, mention it to the patch owner. It's better to have a conversation than to quietly fret over patches
  • Be prepared.
Open all your SSH connections and error logs before you start deploying code.

git

  • git can be confusing. It is more confusing when you're trying to accomplish something under pressure. There are a few config options that might be helpful.
  1. Use a git-aware prompt.
    The git contrib git-prompt script is available on deployment servers. There are instructions for use in comments at the beginning of the file. One simple way to use it is to add the following to your ~/.profile:
    GIT_PS1_SHOWUNTRACKEDFILES=1
    GIT_PS1_SHOWDIRTYSTATE=1
    GIT_PS1_SHOWUPSTREAM="auto verbose"
    . /etc/bash_completion.d/git-prompt
    PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '
    
  2. status.submoduleSummary
    Submodules have limited visibility in git status and it's easy to miss the git submodule update step. After adding status.submoduleSummary to your ~/.gitconfig git will show you a short summary of submodule changes in git status. Set it by executing:
    you@deploy1001:~$ git config --global status.submoduleSummary true
    

SSH Connections and Error Logs

When running SWAT, it is helpful to watch error logs as you deploy so that you can be sure nothing you have just deployed is broken. Also, there are several machines on which you may need to run commands depending on the nature of the SWAT; it's good to open all SSH connections before you have to think about them.

Browser tabs

Terminal tabs

you@laptop$ ssh mwlog1001.eqiad.wmnet

you@mwlog1001:~$ fatalmonitor
you@laptop$ ssh deployment.eqiad.wmnet

you@deploy1001:~$ cd /srv/mediawiki-staging
you@laptop$ ssh mwdebug1002.eqiad.wmnet

you@mwdebug1002:~$ scap pull
you@laptop$ ssh mwmaint1002.eqiad.wmnet

you@mwmaint1002:~$

Merging and applying patches

For SWAT, you will be merging patches either for a wmf branch of mediawiki repositories (core, or a WMF-installed extension or skin), or the operations/mediawiki-config repo.

Merging patches

When +2ing patches, it's often helpful to have the Zuul Dashboard open

  • to ensure that Zuul is picking up your changes,
  • to see how long (approximately) the test will take before it auto-merges.

It's a good practice to put SWAT as the comment when you +2 before you click Publish Comments to ensure that there is a record of why you approved the patch.

Fetching patches

After the patch has merged in Gerrit, you need to fetch it down to deployment.eqiad.wmnet. Make sure that the code you fetch down to deployment.eqiad.wmnet is the code you expected to fetch down.

Use git log -p HEAD..@{u} after you git fetch to check that the patch(es) you fetched down were the same ones the you +2'd. If they aren't, poke the person that wrote the patch in #wikimedia-operations connect to figure out what to do with the fetched code. It's always better to ask than to do something silently and unilaterally.

Process is slightly different for operations/mediawiki-config, mediawiki/core, mediawiki/extensions and mediawiki/skins.

operations/mediawiki-config

you@deploy1001:/srv/mediawiki-staging$ git status

you@deploy1001:/srv/mediawiki-staging$ git fetch

you@deploy1001:/srv/mediawiki-staging$ git log -p HEAD..@{u}

you@deploy1001:/srv/mediawiki-staging$ git rebase

mediawiki/core

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git status

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git fetch

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git log HEAD..@{u}

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git rebase

mediawiki/extensions and mediawiki/skins

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]/[extensions|skins]/[NAME]$ git status

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]/[extensions|skins]/[NAME]$ git fetch

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]/[extensions|skins]/[NAME]$ git log HEAD..origin/wmf/[VERSION]
you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git status

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git fetch

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git log -p HEAD..@{u}

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git rebase

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git status [extensions|skins]/[NAME]

you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git submodule update [extensions|skins]/[NAME]

Deploying changes

Canary

After changes have been fetched and otherwise git-wrangled on deployment.eqiad.wmnet, changes can be fetched down to mwdebug1002.eqiad.wmnet and tested via the X-Wikimedia-Debug header.

you@mwdebug1002:~$ scap pull

After changed have been fetched, ask patch-owner to test changes on mwdebug1002.eqiad.wmnet using X-Wikimedia-Debug.

Full deployment

After a change has been tested on mwdebug1002.eqiad.wmnet it can be deployed to all machines. To deploy the code you will run: scap sync-file <file> [message for SAL]. The code path passed to scap sync-file should be relative to /srv/mediawiki-staging.

The message you type after the file or directory name to be synced will appear in the Server Admin Log — wikitext is legal and can be useful. Copy/pasting the wikitext for that SWAT item from the Deployments calendar is easy. If the Gerrit change has an associated Phabricator task, mention the task ID in the message as appropriate. This will trigger Stashbot to reply back on tasks and indicate that the associated change was synced.

you@deploy1001:/srv/mediawiki-staging$ scap sync-file [FILE|FOLDER] 'SWAT: [[gerrit:[GERRIT-NUMBER]|[COMMIT-MESSAGE] ([PHABRICATOR-TASK])]]'

operations/mediawiki-config

Example:

you@deploy1001:/srv/mediawiki-staging$ scap sync-file wmf-config/InitialiseSettings.php 'SWAT: [[gerrit:444901|Enable FileExporter for sourceswiki (T198594)]]'

mediawiki/core

Example:

you@deploy1001:/srv/mediawiki-staging$ scap sync-file php-1.32.0-wmf.12/includes/cp/ChronologyProtector.php 'SWAT: [[gerrit:445113|rdbms: fix value of ChronologyProtector::POSITION_COOKIE_TTL ([T194403])]]'

mediawiki/extensions and mediawiki/skins

Example:

you@deploy1001:/srv/mediawiki-staging$ scap sync-file php-1.32.0-wmf.12/extensions/WikimediaEvents 'SWAT: [[gerrit:445377|Add event logging for WMDE fundraising banners (T197571)]]'

Purging

See also Multicast HTCP purging#One-off purge.

When a patch for mediawiki-config changes a file in /static, its public url must be purged from Varnish. For performance reasons, unversioned files in static have unconditional caching for up to a year. They rely on manual purges to propagate updates. This purge must be done with en.wikipedia.org as hostname of the url, regardless of which wiki the file relates to. This is because the cache for /static is shared between all wikis, and the canonical form internally for it is en.wikipedia.org.

you@mwmaint1002:~$ echo "https://en.wikipedia.org/static/images/project-logos/newikibooks.png" | mwscript purgeList.php
  • Refresh url in browser.

Verification

After the sync and any purge is finished, monitor logs and ask patch-owner to, once again, test their changes to confirm the patch was deployed successfully. Make sure the patch-owner verifies it with X-Wikimedia-Debug turned off.

Reverting

If a patch doesn't work as expected, or causes errors, it will have to be reverted.

Process

Revert commit causing errors

you@deploy1001:[FOLDER]$ git revert [SHA1]

If the patch being reverted is a merge commit you will have to supply -m

you@deploy1001:[FOLDER]$ git revert [SHA1] -m1

Push code live BEFORE pushing patches to Gerrit

you@deploy1001:/srv/mediawiki-staging$ scap sync-file [FILE/FOLDER] 'SWAT: [[gerrit:[NUMBER]|Revert "[TEXT]" (T[NUMBER])]]'

Push revert patch to Gerrit

you@deploy1001:[FOLDER]$ git push origin HEAD:refs/for/[BRANCH]/revert-[SHA1]

You will be prompted for your Gerrit https password.

Maintenance scripts

During the course of SWAT, you may encounter a patch that needs a maintenance script to be run as part of deployment. As noted earlier, maintenance scripts are run from mwmaint1002.eqiad.wmnet or mwmaint2001.codfw.wmnet.

For convenience, the most frequently run maintenance scripts are presented below.

tmux or screen

For long running scripts, it is recommended they are run in tmux or screen.

If you prefer tmux:

you@mwmaint1002:~$ tmux new -s 'swat'
...
you@mwmaint1002:~$ exit

If you need to leave in the middle you can do ctrl-b d to detach.

If you prefer screen:

you@mwmaint1002:~$ screen -D -RR swat
...
you@mwmaint1002:~$ exit

If you need to leave in the middle you can do ctrl-a d to detach.

createExtensionTables

Allow to source SQL files to create tables for most extensions we deploy.

For example, to create on ar.wikipedia PageAssessments tables:

you@mwmaint1002:~ $ mwscript extensions/WikimediaMaintenance/createExtensionTables.php arwiki pageassessments

namespaceDupes

When a new namespace is added to an existing wiki, the namespaceDupes maintenance script should be run for that wiki. First do a dry run of namespaceDupes for the wiki (without --fix) as a sanity check. Then append --fix to fix namespace duplication:

you@mwmaint1002:~ $ mwscript namespaceDupes.php testwiki
you@mwmaint1002:~ $ mwscript namespaceDupes.php testwiki --fix

updateCollation

When the default collation is changed for a wiki, the updateCollation.php maintenance script will need to be run:

you@mwmaint1002:~$ mwscript updateCollation.php --wiki=iswiki --previous-collation=<VALUE>

Replace <VALUE> with what the wiki's previously configured collation name was in $wgCategoryCollation.

The default collation in MediaWiki is uppercase, as such in most cases when a wiki switches to a different collation the previous can be specified as --previous-collation=uppercase.

resetAuthenticationThrottle

See Increasing account creation threshold.