SWAT deploys

From Wikitech
(Redirected from Lightning deployments)
Jump to navigation Jump to search
Putting the prod in production. Production Drive Committee - NARA - 534919.jpg

SWAT deploys are deployments that happen during a SWAT deploy window (naturally) that are done by a member(s) of the SWAT deploy team (see below).

The purpose is to provide a known window for people to get bug fixes deployed ahead of the normal cadence (currently weekly) without having to beg people to do the deployment for them; the SWAT team is there to do the deployment part.


  • There will be at least one SWAT deploy team member available and active during the window.
    • If no SWAT team member is around and available to do the deploys the window will be skipped. Please reschedule your patches.
  • All those submitting patches for deployment MUST be in #wikimedia-operations connect on Freenode to communicate with the SWAT team member.
    • The SWAT team will ping the relevant developers at the start of the window and when theirs is up; they MUST be available. If they are not available the patch will NOT be deployed.
  • All communication MUST happen in #wikimedia-operations connect on Freenode (not in separate team or area-specific channels)
  • Allowed types of patches Things not fitting these criteria should instead use the standard deploy window process
    • For mediawiki/core and mediawiki/extensions:
      • Fixes of regressions
      • Backports only (IOW: Everything should already be committed into master and tested on mw:Beta Cluster and then "backported" to the relevant release branch)
    • For operations/mediawiki-config
      • Config changes (including enabling/disabling features)
      • If it's complicated or risky, please request a special window or to synchronize with the weekly train.
  • Forbidden types of patches
    • Single patches that require more than one sync - in other words, changes to multiple files which depend on each other.
      • Instead, please break up the patches into multiple safe patches that can be deployed by themselves. See: Task T187761
    • No new extensions
    • Nothing that still needs prior public communication with affected wikis (this is subjective at times, and the SWAT team reserves the right to not deploy if they feel uncomfortable)
  • The SWAT team may ask questions regarding the patches to understand the implications and assess risk. The relevant developers should ideally be on IRC in the hour prior to the SWAT window.
  • The SWAT team MUST be comfortable with the patch going out and CAN veto any proposed patch they are not comfortable with for ANY reason.
  • Our windows have a limit of 6 patches.
    • NOTE: Cherry-picking a patch to both release branches counts as 2 as they will be separate deployments.

How to submit a patch for SWAT

  1. If it is a backport to a branch
    1. Commit the fix in master first
    2. Test that the issue is truly fixed on the mw:Beta Cluster (if possible)
    3. Prepare patches in Gerrit against the current live branches (named wmf.NN) (or a subset if the bug is limited)
      1. Depending on the patch, positive reviews beforehand are necessary (the SWAT team is not responsible for code review)
  2. If it is a config change
    1. Prepare patches in Gerrit against the master branch of the mediawiki-config repo
  3. Add the gerrit URL and your IRC name to Deployments calendar page in the correct SWAT deploy slot.
  4. Be sure that the person whose name appears on the Deployments calendar page will be present for the deployment and able to test the patch, especially if it is a different person from the author of the patch.
Example entry:
* [wmf.8] {{gerrit|297697}} Make LocalRename jobs run sequentially
* [wmf.9] {{gerrit|297698}} Make LocalRename jobs run sequentially
* [config] {{gerrit|431759}} Remove unused PopupsAnonsExperimentalGroupSize config variable

Doing the deploy


  • The SWAT team coordinates the merging and deploying of the patches. The order to deploy the patches is decided by them.
  • The relevant developers should have their test cases ready to run as soon as their patches are deployed.

The process:

  1. The deploy IRC bot (jouncebot) will announce the start of the window and ping the SWAT team along with anyone who has submitted a patch to deploy.
  2. The SWAT team member reviews the patches and picks the ordering.
  3. The SWAT team member identifies the patch to merge, asks if the submitter is ready to test, and merges the backport(s)
  4. After merge, the SWAT team member fetches the patch(es) to deploy1001.eqiad.wmnet and then runs scap pull on a mwdebug host (typically mwdebug#002)
  5. The submitter tests the change by using the instructions at X-Wikimedia-Debug#Staging_changes AND the SWAT team member checks the error logs
  6. If there are no errors and the fix seems to work (if testable in that manner), then SWAT team member deploys the patch to the entire fleet
  7. The submitter tests again (without X-Wikimedia-Debug) AND the SWAT team member checks the logs again.
  8. If everything is good, the next patch is selected and the process starts again.

SWAT Team members' roles, responsibilities, and tips


  • Being a member of the SWAT team imparts a large amount of trust on the person. In some ways more trust that simply access to deploying on the Wikimedia cluster as others are encouraged to ask you to deploy things on their behalf and you must be willing to say "No" when you are uncomfortable. Making mistakes is to be human, but not learning from them will cause SWAT deployers to lose their deployment access.


  • SWAT deployers need not be experts in all parts of our infrastructure, but they must be comfortable with assessing the general risk of a given patch. If needed, they should ask probing questions to the developer submitting the patch to learn more.
    • Experience with MediaWiki and MediaWiki config changes a plus as that is the vast majority of changes that come through the SWAT process.
  • Some unintuitive situations include:
    • a "simple" config change causing a load spike in a dependent system that the deployer or developer is not familiar with
    • a "simple" config change being against "the community's", the Wikimedia Foundation's, or both's desires
      • controversial changes can easily be skipped and referred to the WMF Release Manager for next steps, there is no need to rush these
  • If a SWAT deployer is uncomfortable with a certain area of the code-base they are free to skip that backport at their own discretion (or have another SWAT deployer review and/or deploy it).


  • SWAT deployers should not feel obligated to help a developer debug a situation, especially if there is a user-facing issue or outage.
  • When in doubt: Revert and ask questions later

New SWAT Team member check-list

  • Read and be comfortable with the above roles, responsibilities and tips
  • Shell and deploy access in production, see
  • Access to merge changes in wmf deploy branches by being added to the wmf-deployments gerrit group
    • Ask any existing wmf-deployments group member to do this.
  • Join (and read) the operations mailing list (ops@lists.wikimedia.org)
    • This is because announcements that could impact how and/or when to deploy things are primarily sent there.
  • Read the docs:


Artist rendering of a blazing wiki

SWAT, or the Setting Wikis Ablaze Team, is responsible for breaking the site on a regular basis so we all don't get too used to a stable platform upon which to develop. Compare chaos monkey or unit testing (which are similar to each other) - this is less automatic but much more effective at breaking things. The term disintegration testing is also used for this, named after how the Mars Climate Orbiter was tested.