User:Thcipriani/Scap3 MediaWiki

This was a conversation during which we discussed bringing scap3-like functionality to MediaWiki. It informed the Q2 Goals for the Release Engineering team, but nothing here is set in stone.

These notes also may be completely meaningless to folks who were not in that meeting.

Q2–Q4 Goals Scap/Deployment/Mediawiki

Random Notes

• Version number for ALL deployed mediawiki code (trust deploymented machines)
  • How do static assets fit into this?
  • If it's on disk, it's in there

• "Scap 2.5" A.K.A Definition of done
  • Not necessary to clean and merge all code paths of scap2 and scap3
  • Meet Certain criteria instead:
    • Single command to deploy all code: scap push [/path/to/code OR directory]
    • Atomic deployments, no longer have to worry about sync order
    • Definable checks at each stage

• Here's a outline of one possible way to add git transport to the current scap system:
  • https://phabricator.wikimedia.org/T101023#1349229

• Scap 3.1
  • Split out deploy and deploy-local stages into individual classes per stage
  • ability to override stages by subclassing
  • implement mediawiki scap deployment on top of scap 3 with some customizations in subclasses

Stretch goals

• l10nupdate ...
• Depool proxies temporarily while scap is ongoing to avoid taxing those nodes https://phabricator.wikimedia.org/T125629
• Translation cache exhaustion caused by changes to PHP code in file scope https://phabricator.wikimedia.org/T103886
  • De/Repool and restart

Big Hairy Audacious Plan (BHAP)

On Tin

1. tin.eqiad.wmnet: rsync --exclude=**/.git /srv/mediawiki-staging → /srv/mediawiki
2. tin.eqiad.wmnet: git -C /srv/mediawiki init

On Targets

1. git -C /srv/mediawiki-cache/ pull
2. git archive /srv/mediawiki-cache /srv/mediawiki-cache/revs/[SHA1]
3. ln -sfT /srv/mediawiki-cache/revs/[SHA1] /srv/mediawiki
4. ???
5. profit

Subpoints

• Use rsync from tin for the initial creation of /srv/mediawiki-cache—not going to beat rsync on the network
• Disk IO no longer (likely) the anchor, network IO is—how bad is this change for the network?
• .gitignore /srv/mediawiki/php-*/cache/l10n/*.cdb

By Quarter

strawman

• BHAP (using a git repo for MW deploys)
  • using current rsync solution
• git syncing a stretch goal
• assess new directory structure and how it will interplay with MW
• gives us:
  • single version number
  • a git repo to use for scap3

Q2

• use a git repo for MW deploys
• assess new directory structure and how it will interplay with MW
• Stretch goal: git syncing

Q3

• build fanout in scap3
• assess impact of moving to `scap deploy` (scap3)

Q4

• migrate to scap3
• Integrate scap with etcd/pybal to automatically depool and repool servers - task T104352

smaller, shaved yak plan

• build the git repo on tin
  • /srv/mediawiki becomes a flattened git repo
  • continue using rsync to deploy this

• figure out how to deploy this with scap3
  • this involves working out scap3 fanout