Jump to content

Backport windows

From Wikitech
(Redirected from Lightning deployments)

Backport windows are set times for Wikimedia production deployments, are done by a member(s) of the backport deploy team (see below). Despite the name, they are used for configuration changes to production wikis as well as backports.

The purpose is to provide a known window for people to get bug fixes deployed ahead of the normal cadence (currently, weekly) without having to beg people to do the deployment for them; the team is there to do the deployment part. Similarly, it is usually advantageous to make configuration changes separate from train deploys so that effects of the reconfiguration can be distinguished from regressions caused by the train; the backport deploy team provides the expertise to safely deploy configuration changes.

They were historically known as SWAT deploy windows.

The words MUST, CAN, MAY, etc. are to be interpreted in conformity with RFC:2119.

Guidelines

  • There will be at least one backport window deployer available and active during the window.
    • If no backport team member is around and available to do the deploys the window will be skipped. Please reschedule your patches.
  • All those submitting patches for deployment MUST be in #wikimedia-operations connect on libera.chat to communicate with the backport team member.
    • The backport team will ping the relevant developers at the start of the window and when theirs is up; they MUST be available. If they are not available the patch will NOT be deployed.
  • All communication MUST happen in #wikimedia-operations connect on libera.chat (not in separate team or area-specific channels)
  • Allowed types of patches Things not fitting these criteria should instead use the standard deploy window process
    • For mediawiki/core and mediawiki/extensions:
      • Fixes of regressions
      • Backports only (In other words, everything should already be committed into master and tested on mw:Beta Cluster and then "backported" to the relevant release branch)
    • For operations/mediawiki-config
      • Config changes (including enabling/disabling features, refactoring, etc)
      • Some changes may require their own Deployment Window.
  • Forbidden types of patches
    • Single patches that require more than one sync
      • Every sync restarts PHP, so cross-file PHP dependencies are fine
      • Be careful with changes for extension.json—it is loaded dynamically, outside the PHP restart
    • No new extensions
    • Nothing that still needs prior public communication with affected wikis (this is subjective at times, and the backport team reserves the right to not deploy if they feel uncomfortable)
  • The backport team may ask questions regarding the patches to understand the implications and assess risk. The relevant developers should ideally be on IRC in the hour prior to the backport window.
  • The backport team MUST be comfortable with the patch going out and CAN veto any proposed patch they are not comfortable with for ANY reason.
  • Our windows have a limit of 6 patches.
    • NOTE: Cherry-picking a patch to both release branches counts as 2 as they will be separate deployments.

How to submit a patch for backport

  1. Prepare the patch in Gerrit.
    • If it is a backport to a branch:
      1. Commit the fix in master first, and get it reviewed and merged.
      2. Test that the issue is truly fixed on the mw:Beta Cluster (if possible)
      3. Identify what branch is rolling out (the leftmost branch).
      4. Prepare patches in Gerrit against the current live branches (named wmf.NN) (or a subset if the bug is limited). You can let Gerrit create the cherry-pick, using the “Cherry pick” option under the ⋮ menu on the original patch. Enter the target branch, and leave the commit message as it is. (If that doesn't work, see #Submitting a manual cherry-pick below.)
    • If it is a config change:
      1. Prepare patches against the master branch of the operations/mediawiki-config repo.
  2. Add the gerrit URL and your IRC name to Deployments calendar page in the correct backport deploy slot.
    • You can use the backport scheduling tool to more easily edit this page.
    • Alternatively, if you are manually editing the page, double check the time, that the slot says "backport", your IRC handle, and the patch number.
  3. Double check that you have the WikimediaDebug extension installed. You'll need this to verify the change after its been deployed. (As of April 2024, this means you will have to use a desktop browser.)
  4. Be sure that the person whose name appears on the Deployments calendar page will be present on the #wikimedia-operations connect IRC channel for the deployment and able to test the patch, especially if it is a different person from the author of the patch. Now is a good time to double check your IRC nick and connection. If you've done everything right, jouncebot will ping you when your backport window opens. This is a good time to ping the scheduled deployers directly.

Example entry:

{{ircnick|legoktm}}
{{deploy|type=1.28.0-wmf.8|gerrit=297697|title=Make LocalRename jobs run sequentially|status=}}
{{deploy|type=1.28.0-wmf.9|gerrit=297698|title=Make LocalRename jobs run sequentially|status=}}
{{deploy|type=config|gerrit=431759|title=Remove unused PopupsAnonsExperimentalGroupSize config variable|status=}}

Submitting a manual cherry-pick

If you need to backport a chain of patches, or the Gerrit cherry-pick doesn’t work for some reason, you can do it manually:

  • For example, if the branch is 1.35.0-wmf.11 and the patch's Git hash is ec56a606, execute git fetch origin wmf/1.35.0-wmf.11 && git checkout wmf/1.35.0-wmf.11 && git cherry-pick ec56a606. The latter cherry-pick command can also be found in the Gerrit old UI under the download menu. E.g., git fetch "https://gerrit.wikimedia.org/r/mediawiki/skins/MinervaNeue" refs/changes/23/559223/1 && git cherry-pick FETCH_HEAD.
  • Now push the patch up. Assuming the branch is 1.35.0-wmf.11, git push origin HEAD:refs/for/wmf/1.35.0-wmf.11. If Git wants a password, it's your HTTP password found in the Gerrit settings.

Doing the deploy

Generally:

  • The backport team coordinates the merging and deploying of the patches. The order to deploy the patches is decided by them.
  • The relevant developers should have their test cases ready to run as soon as their patches are deployed.

The process:

  1. The deploy IRC bot (jouncebot) will announce the start of the window and ping the backport team along with anyone who has submitted a patch to deploy.
  2. The backport team member reviews the patches and picks the ordering.
  3. The backport team member identifies the patch to merge, asks if the submitter is ready to test, and merges the backport(s)
  4. After merge, the backport team member fetches the patch(es) to deploy1002.eqiad.wmnet and then runs scap pull on a mwdebug host (typically mwdebug#002)
  5. The submitter tests the change by using the instructions at X-Wikimedia-Debug#Staging_changes AND the backport team member checks the error logs
  6. If there are no errors and the fix seems to work (if testable in that manner), then the backport team member deploys the patch to the entire fleet
  7. The submitter tests again (without X-Wikimedia-Debug) AND the backport team member checks the logs again.
  8. If everything is good, the next patch is selected and the process starts again.
For a more detailed outline of the steps above refer to the Backport deployers'-specific documentation.

Backport Team members' roles, responsibilities, and tips

Trust

  • Being a member of the backport team imparts a large amount of trust on the person. In some ways more trust that simply access to deploying on the Wikimedia cluster as others are encouraged to ask you to deploy things on their behalf and you must be willing to say "No" when you are uncomfortable. Making mistakes is to be human, but not learning from them will cause backport deployers to lose their deployment access.

Knowledge

  • Backport deployers need not be experts in all parts of our infrastructure, but they must be comfortable with assessing the general risk of a given patch. If needed, they should ask probing questions to the developer submitting the patch to learn more.
    • Experience with MediaWiki and MediaWiki config changes a plus as that is the vast majority of changes that come through the backport process.
  • Some unintuitive situations include:
    • a "simple" config change causing a load spike in a dependent system that the deployer or developer is not familiar with
    • a "simple" config change being against "the community's", the Wikimedia Foundation's, or both's desires
      • controversial changes can easily be skipped and referred to the WMF Release Manager for next steps, there is no need to rush these
  • If a backport deployer is uncomfortable with a certain area of the code-base they are free to skip that backport at their own discretion (or have another backport deployer review and/or deploy it).

Decisiveness

  • Backport deployers should not feel obligated to help a developer debug a situation, especially if there is a user-facing issue or outage.
  • When in doubt: Revert and ask questions later

New Backport Team member check-list

  • Read and be comfortable with the above roles, responsibilities and tips
  • Shell and deploy access in production, see
  • Access to merge changes in wmf deploy branches by being added to the wmf-deployments gerrit group
    • Ask any existing wmf-deployments group member to do this.
  • Join (and read) the operations mailing list (ops@lists.wikimedia.org)
    • This is because announcements that could impact how and/or when to deploy things are primarily sent there.
  • Read the docs:

Subpages

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Backport_windows&oldid=2245909"