Nova Resource:Deployment-prep/How code is updated

The cluster runs the master branch of MediaWiki core and extensions.

The extensions are deployed using the Gerrit project mediawiki/extensions.git in which extensions are manually registered as submodules. The Jenkins job beta-code-update-eqiad runs on the deployment host within the deployment-prep project to update MediaWiki core and all the submodules under /extensions. The job then triggers a job to run scap sync-world and starts over again every ten minutes.

An overview of all the jobs for this project is available under the Jenkins 'Beta' view. As of June 2025 the jobs are:

beta-code-update-eqiad (jenkins): Update MediaWiki core and extensions to their master, and trigger the beta-scap-sync-world job every 10 minutes.

beta-scap-sync-world (jenkins): Run scap sync-world to update localization cache and copy MediaWiki core, extensions and settings to the rest of the beta cluster when triggered by a successful beta-code-update-eqiad job.

beta-update-databases-eqiad (jenkins): Job to run through all databases and run the update.php script against each of them on a hourly basis.

The job status can be seen on the Wikimedia Continuous integration portal https://integration.wikimedia.org/ or by looking at the Beta Jenkins view https://integration.wikimedia.org/ci/view/Beta/

Puppet

We are using a project puppetserver in the project.

Converting a host to use local puppetserver

See Help:Project puppetserver#Step 2: Setup a puppet client for manual steps that may be needed to finish the process of switching from the Cloud VPS shared Puppetserver to the deployment-prep local Puppetserver. There is global hiera config in deployment-prep that will start the process of switching (step 1 in the linked documentation). The parts for handling certificate verify failed: [self signed certificate in certificate chain ...] failures are what may need to be manually performed.

Updating puppet manifests to match production

To force an immediate update of the latest puppet configuration you will need to log into deployment-puppetserver-1 and update /srv/git/operations/puppet manually before forcing the puppet run. We've been doing this:

ssh deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
sudo -s su gitpuppet
exec /bin/bash
cd /srv/git/operations/puppet
git fetch
git rebase origin/production

Cherry-picking a patch from gerrit

The major benefit of using the local puppet master in the beta project is the ability to quickly add and test new puppet configuration before getting someone with +2 in operations/puppet.git to merge. This is often needed because an ugly temporary fix is needed for beta while a nicer long term solution is worked out for production.

We use cherry-picks rebased on top of the current operations/puppet.git production branch because git is smart about these sorts of changes and will recognize on subsequent git rebase runs patches that have been merged upstream.

Create a patch against operations/puppet.git
git review
Go to review in gerrit and find the "cherry-pick" git command that gerrit generates for the patch. It will look something like: git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/88/125888/1 && git cherry-pick FETCH_HEAD
ssh deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
sudo su - gitpuppet -c 'exec /usr/bin/bash' (this prints harmless errors about ioctl and job control)
cd /srv/git/operations/puppet
git fetch
git rebase origin/production
- This may print some errors like Current branch in /srv/git/operations/puppet is "", should be "production", this is normal
Paste cherry-pick command from gerrit: git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/... && git cherry-pick FETCH_HEAD
- Occasionally there will be a conflict resulting from the cherry-pick; figure out how to resolve it if that happens or git cherry-pick --abort to roll back the attempt
- You may find that you need to go back to your local working copy and rebase your patch on top of another cherry-pick that is already pending review in gerrit to cleanly resolve the conflict.
Ssh to a host that the patch should effect and force a puppet run with sudo run-puppet-agent
- If puppet is not picking up your changes, you may need to run sudo /usr/local/bin/puppetserver-deploy-code as gitpuppet on the puppetserver so that it picks up your cherry-picked changes. This should run as a hook, but hooks can fail.
Log the change (i.e. go to #wikimedia-releng ^connect and use "!log").

If testing a temporary patch remember to remove the cherry-pick after you are done testing (see how-to below).

If the patch is not a temporary test, but something needed for Beta Cluster to work properly, then:

Add hash tag "beta-cherry-picked" to the Gerrit change.
Verify that it passes Puppet-compiler with a set of relevant hosts (if triggered manually, linked to your result in a Gerrit comment).
Report back your test to Gerrit in a comment, and ask SRE to merge it. Mention that it's live on Beta Cluster and working as intended.

Updating/removing a cherry-pick

When you find a bug in your proposed patch (it happens to the best of us), you'll want to remove the old cherry-picked patch and possibly add an amended one.

ssh deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
sudo -s su gitpuppet
exec /bin/bash
cd /srv/git/operations/puppet
git rebase --interactive
Find the line for the cherry-pick you want to remove/update and delete it from the git-rebase-todo file that opens in your editor
Save the file/close your editor (:q in vim; you do use vim right?)
Cross you fingers that the rebase completes with no errors
- Clean up git with git rebase --abort if there are conflicts you can't figure out how to resolve.

If you wanted to update the cherry-pick, go back to the Cherry-picking a patch from gerrit instructions now that the old cherry-pick is removed.

FAQ

How do I get my code on the beta cluster?

To get your code deployed, you just have to have it merged in the master branch. It will eventually land on beta after a few minutes. Have a look at the Jenkins beta-code-update-eqiad job.

How does MediaWiki configuration get updated?

We use a Jenkins job that reacts on merges happening in operations/mediawiki-config.git. That triggers a Jenkins job beta-mediawiki-config-update-eqiad which connects to the bastion instance and checkout the merged revision. The job invalidates the cached configuration by touching InitialiseSettings.php.

How does the database get upgraded?

If your change is supported by MediaWiki update.php, it will be applied via the hourly Jenkins job beta-update-databases-eqiad. The job spawn a child job for each of the databases. They will simply run update.php.

How to add a new extension on beta

At first, beta is a staging area, it is not meant to be a development playground. The extension you want to add to beta must have been planned for deployment on production. If you want a dev sandbox, you should create an instance in a different labs project.

Steps to deploy a new extension on beta cluster:

Make sure you have tests registered in Jenkins / Zuul. You don't want to deploy a php linting error
Verify the extension is registered as a WMF-branched extension in make-wmf-branch (found in repos/releng/release)
Register the extension in operations/mediawiki-config.git
- Add entry point in /wmf-config/extension-list (used to generate l10n messages)
- Add a wfLoadExtension() in CommonSettings.php
Eventually create a global $wmgUse<your extension name> including the default of off/false in InitialiseSettings.php
Create settings using either CommonSettings-labs.php or InitialiseSettings-labs.php

Whenever your change to operations/mediawiki-config.git is merged, it will be deployed by Jenkins job beta-mediawiki-config-update-eqiad as described above.

Remember: you don't want the extension to be enabled if it is not ready / added to wmf branches yet.

My code introduces a feature that is not yet ready for production, should I wait to merge in master?

You definitely do not want your experiment to land on production which would potentially cause a lot of havoc. A best practice we have been following for more than a decade is to safe guard your new feature using a MediaWiki setting. Having a setting set to false by default, would make sure the feature is not going to be enabled on production. You could then explicitly enable it on beta by editing the wmf-config/InitialiseSettings-labs.php file of operations/mediawiki-config.git.

Example highlighting how to safeguard a new feature (see mw:Manual:Configuration for developers for more details):

Add the flag to extension.json (assuming you’re in an extension) with the default value:

	"config": {
		"MyExtensionEnableFeatureOne": {
			"value": false
		}
	}

Check it in your PHP code (optionally, you can inject the config as well, or even use the ConfigFactory):

function something() {
	$config = MediaWikiServices::getInstance()->getMainConfig();
	if ( $config->get( 'MyExtensionEnableFeatureOne' ) ) {
		// Your new code
	}
	// some more code
}

Then to enable this feature on beta:

// wmf-config/InitialiseSettings-labs.php under wmfLabsSettings()
  'wgMyExtensionEnableFeatureOne' => [ 'default' => true ],

Get it merged. Jenkins deploys the new configuration, the feature is enabled \O/