SWAT deploys

From Wikitech
Jump to: navigation, search
Putting the prod in production. Production Drive Committee - NARA - 534919.jpg

SWAT deploys are deployments that happen during a SWAT deploy window (naturally) that are done by a member(s) of the SWAT deploy team (see below).

The purpose is to provide a known window for people to get bug fixes deployed ahead of the normal cadence (currently weekly) without having to beg people to do the deployment for them; the SWAT team is there to do the deployment part.


  • There will be at least one SWAT deploy team member available and active during the window.
  • All those submitting patches for deployment MUST be in #wikimedia-operations connect on Freenode to communicate with the SWAT team member.
    • The SWAT team will ping the relevant developers at the start of the window and when theirs is up; they MUST be available. If they are not available the patch will NOT be deployed.
  • All communication MUST happen in #wikimedia-operations connect on Freenode (not in separate team or area-specific channels)
  • Allowed types of patches Things not fitting these criteria should instead use the standard deploy window process
    • Fixes of regressions
    • Simple config changes (that don't turn on any new features)
    • Everything should be already committed into master and tested on mw:Beta Cluster and then backported to the relevant release branch
  • Forbidden types of patches
    • No new features/extensions
    • Nothing that needs prior public communication with affected wikis (this is subjective at times, and the SWAT team reserves the right to not deploy if they feel uncomfortable)
  • The SWAT team may ask questions regarding the patches to understand the implications and assess risk. The relevant developers should ideally be on IRC in the hour prior to the SWAT window.
  • The SWAT team MUST be comfortable with the patch going out and CAN veto any proposed patch they are not comfortable with for ANY reason.
  • Our windows have a limit of 8 patches.
    • If you're cherry-picking a patch to both release branches, that counts as 2. Speak to the Foundation's Release Manager (Greg Grossmeier aka "greg-g") in #wikimedia-operations connect on Freenode

How to submit a patch for SWAT

  1. Commit the fix in master first
  2. Test that the issue is truly fixed on the mw:Beta Cluster (if possible)
  3. Prepare patches in Gerrit against the current live branches (named wmf.NN) (or a subset if the bug is limited)
    1. Depending on the patch, positive reviews beforehand are necessary (the SWAT team is not responsible for code review)
  4. Add the gerrit URL and your IRC name to Deployments calendar page in the correct SWAT deploy slot.
  5. Be sure that the person whose name appears on the Deployments calendar page will be present for the deployment and able to test the patch, especially if it is a different person from the author of the patch.
Example entry:
* [wmf.8] {{gerrit|297697}} Make LocalRename jobs run sequentially
* [wmf.9] {{gerrit|297698}} Make LocalRename jobs run sequentially

Doing the deploy


  • The SWAT team coordinates the merging and deploying of the patches. The order to deploy the patches is decided by them.
  • The relevant developers should have their test cases ready to run as soon as their patches are deployed.

The process:

  1. The deploy IRC bot (jouncebot) will announce the start of the window and ping the SWAT team along with anyone who has submitted a patch to deploy.
  2. The SWAT team member reviews the patches and picks the ordering.
  3. The SWAT team member identifies the patch to merge, asks if the submitter is ready to test, and merges the backport(s)
  4. After merge, the SWAT team member fetches the patch(es) to tin and then runs scap pull on mw1099
  5. The submitter tests the change by using the instructions at X-Wikimedia-Debug#Staging_changes AND the SWAT team member checks the error logs
  6. If there are no errors and the fix seems to work (if testable in that manner), then then SWAT team member deploys the patch to the entire fleet
  7. The submitter tests again (without X-Wikimedia-Debug) AND the SWAT team member checks the logs again.
  8. If everything is good, the next patch is selected and the process starts again.

The team

Membership in the SWAT Team is managed by the WMF Release Manager (Greg Grossmeier aka "greg-g" in #wikimedia-operations connect).

06:00 SF window

  • Antoine Musso
  • S├ębastien Santoro (Dereckson)
  • Jan Zerebecki

11:00 SF window

  • Antoine Musso
  • Brad Jorsch
  • Chad Horohoe
  • Katie Filbert
  • Mukunda Modell
  • Tyler Cipriani

16:00 SF window

  • Roan Kattouw
  • Chad Horohoe
  • S├ębastien Santoro (Dereckson)

SWAT Team members roles, responsibilities, and tips


  • Being a member of the SWAT team imparts a large amount of trust on the person. In some ways more trust that simply access to deploying on the Wikimedia cluster as others are encouraged to ask you to deploy things on their behalf and you must be willing to say "No" when you are uncomfortable. Making mistakes is to be human, but not learning from them will cause SWAT deployers to lose their deployment access.


  • SWAT deployers need not be experts in all parts of our infrastructure, but they must be comfortable with assessing the general risk of a given patch. If needed, they should ask probing questions to the developer submitting the patch to learn more.
    • Experience with MediaWiki and MediaWiki config changes a plus as that is the vast majority of changes that come through the SWAT process.
  • Some unintuitive situations include:
    • a "simple" config change causing a load spike in a dependent system that the deployer or developer is not familiar with
    • a "simple" config change being against "the community's", the Wikimedia Foundation's, or both's desires
      • controversial changes can easily be skipped and referred to the WMF Release Manager for next steps, there is no need to rush these
  • If a SWAT deployer is uncomfortable with a certain area of the code-base they are free to skip that backport at their own discretion (or have another SWAT deployer review and/or deploy it).


  • SWAT deployers should not feel obligated to help a developer debug a situation, especially if there is a user-facing issue or outage.
  • When in doubt: Revert and ask questions later

New SWAT Team member check-list

  • Read and be comfortable with the above roles, responsibilities and tips
  • Shell and deploy access in production, see
  • Access to merge changes in wmf deploy branches by being added to the wmf-deployments gerrit group
    • Ask any existing wmf-deployments group member to do this.
  • Join (and read) the operations mailing list (ops@lists.wikimedia.org)
    • This is because announcements that could impact how and/or when to deploy things are primarily sent there.
  • Read the docs:


SWAT, or the Setting Wikis Ablaze Team, is responsible for breaking the site on a regular basis so we all don't get too used to a stable platform upon which to develop. Compare chaos monkey or unit testing (which are similar to each other) - this is less automatic but much more effective at breaking things. The term disintegration testing is also used for this, named after how the Mars Climate Orbiter was tested.