Nova Resource:Deployment-prep/How code is updated

The Beta Cluster runs on the latest master branch of MediaWiki core and extensions. We use more or less the same directory layout and server roles as WMF production. See Heterogeneous deployment for more details.

Code is staged in /srv/mediawiki-staging/ on the deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud instance.
To run a maintenance script, you can run mwscript just as you would in production.
scap commands are run as the user jenkins-deploy.
To become the deploy user, run sudo -u jenkins-deploy -s.
MediaWiki is in /srv/mediawiki-staging/php-master. You can apply unmerged patches by becoming the jekins-deploy user, and running git cherry-pick.
Live hacking MediaWiki code or MediaWiki configuration files on the Beta Cluster can be unpredictable, because these are overwritten by automatic upates every 5 minutes.

Automatically updated

Every 5 minutes, the wmf-beta-update-all systemd timer on the active Beta deployment host updates MediaWiki (core + extensions) and wmf-config to the latest master branch from their Gerrit repositories. The extensions are specifically deployed via the mediawiki/extensions.git meta repository, where new extensions are manually added as submodule, which Gerrit then automatically keeps in sync.

The job started by the timer then runs Scap to deploy the updated MediaWiki code to all MediaWiki hosts in the Beta Cluster, and then runs update.php on all wikis to apply any schema changes.

You can find the status of the update job via the logs at https://beta-update.wmcloud.org.

History

Prior to April 2026, automatic updates were managed via Jenkins. The Beta deployment host was registered as a Jenkins node, and the update scripts were run from Jenkins jobs that were scheduled to run every ten minutes (T256168).

Additionally, a post-merge job for operations/mediawiki-config.git also triggerred the beta-mediawiki-config-update-eqiad job which updated wmf-config almost immediately (instead of waiting ten minutes) followed by a quicker scap sync for just the wmf-config directory.

As of June 2025, the Jenkins jobs were:

beta-code-update-eqiad: Update MediaWiki core and extensions to their master, and trigger the beta-scap-sync-world job every ten minutes.
beta-scap-sync-world: Run scap sync-world to update localization cache and copy MediaWiki core, extensions and settings to the rest of the beta cluster when triggered by a successful beta-code-update-eqiad job.
beta-update-databases-eqiad: Iterate through all wikis and run the update.php maintenance script against each of them on a hourly basis.

Puppet

We are using a project puppetserver in the project.

Converting a host to use local puppetserver

See Help:Project puppetserver#Step 2: Setup a puppet client for manual steps that may be needed to finish the process of switching from the Cloud VPS shared Puppetserver to the deployment-prep local Puppetserver. There is global hiera config in deployment-prep that will start the process of switching (step 1 in the linked documentation). The parts for handling certificate verify failed: [self signed certificate in certificate chain ...] failures are what may need to be manually performed.

Updating puppet manifests to match production

To force an immediate update of the latest puppet configuration you will need to log into deployment-puppetserver-1 and update /srv/git/operations/puppet manually before forcing the puppet run. We've been doing this:

ssh deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
sudo -s su gitpuppet
exec /bin/bash
cd /srv/git/operations/puppet
git fetch
git rebase origin/production

Cherry-picking a patch from gerrit

The major benefit of using the local puppet master in the beta project is the ability to quickly add and test new puppet configuration before getting someone with +2 in operations/puppet.git to merge. This is often needed because an ugly temporary fix is needed for beta while a nicer long term solution is worked out for production.

We use cherry-picks rebased on top of the current operations/puppet.git production branch because git is smart about these sorts of changes and will recognize on subsequent git rebase runs patches that have been merged upstream.

Create a patch against operations/puppet.git
git review
Go to review in gerrit and find the "cherry-pick" git command that gerrit generates for the patch. It will look something like: git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/88/125888/1 && git cherry-pick FETCH_HEAD
ssh deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
sudo su - gitpuppet -c 'exec /usr/bin/bash' (this prints harmless errors about ioctl and job control)
cd /srv/git/operations/puppet
git fetch
git rebase origin/production
- This may print some errors like Current branch in /srv/git/operations/puppet is "", should be "production", this is normal
Paste cherry-pick command from gerrit: git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/... && git cherry-pick FETCH_HEAD
- Occasionally there will be a conflict resulting from the cherry-pick; figure out how to resolve it if that happens or git cherry-pick --abort to roll back the attempt
- You may find that you need to go back to your local working copy and rebase your patch on top of another cherry-pick that is already pending review in gerrit to cleanly resolve the conflict.
Ssh to a host that the patch should effect and force a puppet run with sudo run-puppet-agent
- If puppet is not picking up your changes, you may need to run sudo /usr/local/bin/puppetserver-deploy-code as gitpuppet on the puppetserver so that it picks up your cherry-picked changes. This should run as a hook, but hooks can fail.
Log the change (i.e. go to #wikimedia-releng ^connect and use "!log").

If testing a temporary patch remember to remove the cherry-pick after you are done testing (see how-to below).

If the patch is not a temporary test, but something needed for Beta Cluster to work properly, then:

Add hash tag "beta-cherry-picked" to the Gerrit change.
Verify that it passes Puppet-compiler with a set of relevant hosts (if triggered manually, linked to your result in a Gerrit comment).
Report back your test to Gerrit in a comment, and ask SRE to merge it. Mention that it's live on Beta Cluster and working as intended.

Updating/removing a cherry-pick

When you find a bug in your proposed patch (it happens to the best of us), you'll want to remove the old cherry-picked patch and possibly add an amended one.

ssh deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
sudo -s su gitpuppet
exec /bin/bash
cd /srv/git/operations/puppet
git rebase --interactive
Find the line for the cherry-pick you want to remove/update and delete it from the git-rebase-todo file that opens in your editor
Save the file/close your editor (:q in vim; you do use vim right?)
Cross you fingers that the rebase completes with no errors
- Clean up git with git rebase --abort if there are conflicts you can't figure out how to resolve.

If you wanted to update the cherry-pick, go back to the Cherry-picking a patch from gerrit instructions now that the old cherry-pick is removed.

FAQ

How do I get my code on the beta cluster?

To get your code deployed, you just have to have it merged in the master branch. It will be automatically pulled in by the Beta Cluster within a few minutes. Have a look at https://beta-update.wmcloud.org/.

How to add a new extension on beta

At first, beta is a staging area, it is not meant to be a development playground. The extension you want to add to beta must have been planned for deployment on production. If you want a dev sandbox, you should create an instance in a different labs project.

Steps to deploy a new extension on beta cluster:

Make sure you have tests registered in Jenkins / Zuul. You don't want to deploy a php linting error
Verify the extension is registered as a WMF-branched extension in make-wmf-branch (found in repos/releng/release)
Register the extension in operations/mediawiki-config.git
- Add entry point in /wmf-config/extension-list (used to generate l10n messages)
- Add a wfLoadExtension() in CommonSettings.php
Eventually create a global $wmgUse<your extension name> including the default of off/false in InitialiseSettings.php
Create settings using either CommonSettings-labs.php or InitialiseSettings-labs.php

Whenever your change to operations/mediawiki-config.git is merged, it will be pulled to the Beta Cluster automatically within 5 minutes.

Remember: you don't want the extension to be enabled if it is not ready / added to wmf branches yet.

My code introduces a feature that is not yet ready for production, should I wait to merge in master?

You definitely do not want your experiment to land on production which would potentially cause a lot of havoc. A best practice we have been following for more than a decade is to safe guard your new feature using a MediaWiki setting. Having a setting set to false by default, would make sure the feature is not going to be enabled on production. You could then explicitly enable it on beta by editing the wmf-config/InitialiseSettings-labs.php file of operations/mediawiki-config.git.

Example highlighting how to safeguard a new feature (see mw:Manual:Configuration for developers for more details):

Add the flag to extension.json (assuming you’re in an extension) with the default value:

	"config": {
		"MyExtensionEnableFeatureOne": {
			"value": false
		}
	}

Check it in your PHP code (optionally, you can inject the config as well, or even use the ConfigFactory):

function something() {
	$config = MediaWikiServices::getInstance()->getMainConfig();
	if ( $config->get( 'MyExtensionEnableFeatureOne' ) ) {
		// Your new code
	}
	// some more code
}

Then to enable this feature on beta:

// wmf-config/InitialiseSettings-labs.php under wmfLabsSettings()
  'wgMyExtensionEnableFeatureOne' => [ 'default' => true ],

Once the wmf-config change is merged, it will be live in the Beta Cluster automatically within 5 minutes and your feature is enabled! \O/