Nova Resource:Deployment-prep/How code is updated

The cluster runs the master branch of MediaWiki core and extensions.

The extensions are deployed using the Gerrit project mediawiki/extensions.git in which extensions are manually registered as submodules. The Jenkins job beta-code-update-eqiad runs on the deployment host within the deployment-prep project to update MediaWiki core and all the submodules under /extensions. The job then refreshes the message cache and start again every ten minutes.

An overview of all the jobs for this project is available under the Jenkins 'Beta' view. As of October 2017 the jobs are:

beta-code-update-eqiad (jenkins): Update MediaWiki core and extensions to their master, and trigger the beta-scap-eqiad Jenkins job.

beta-mediawiki-config-update-eqiad (jenkins): Update the working copy of operations/mediawiki-config.git which holds the MediaWiki settings for both production and labs, and trigger the beta-scap-eqiad Jenkins job.

beta-scap-eqiad (jenkins): Run scap to update localization cache and copy MediaWiki core, extensions and settings to the rest of the beta cluster.

beta-update-databases-eqiad (jenkins): Job to run through all databases and run the update.php script against each of them on a hourly basis.

beta-publish-deb (jenkins): Job running on deployment-mira which publishes debian packages built with beta-build-deb. Currently used only by Scap3. Packages are automatically added to a Aptly repository at http://deployment-mira.deployment-prep.eqiad.wmflabs and available from any instance in the project. Simply run apt-get update && apt-get install package-name to install the package from mira. More info: Nova_Resource:Deployment-prep/Beta-Apt-Repo

The job status can be seen on the Wikimedia Continuous integration portal https://integration.wikimedia.org/ or by looking at the Beta Jenkins view https://integration.wikimedia.org/ci/view/Beta/

Puppet

We are using a standalone puppermaster in the project.

Converting a host to use local puppetmaster

This has been automatically set up with hiera and ENC. Just run puppet a few times and wait :)

Updating puppet manifests to match production

To update puppet configuration you will need to log into deployment-puppetmaster04 and update /var/lib/git/operations/puppet manually before forcing the puppet run. We've been doing this:

ssh deployment-puppetmaster04.deployment-prep.eqiad.wmflabs
sudo -s
cd /var/lib/git/operations/puppet
git fetch
git rebase origin/production

Cherry-picking a patch from gerrit

The major benefit of using the local puppet master in the beta project is the ability to quickly add and test new puppet configuration before getting someone with +2 in operations/puppet.git to merge. This is often needed because an ugly temporary fix is needed for beta while a nicer long term solution is worked out for production.

We use cherry-picks rebased on top of the current operations/puppet.git production branch because git is smart about these sorts of changes and will recognize on subsequent git rebase runs patches that have been merged upstream.

Create a patch against operations/puppet.git
git review
Go to review in gerrit and find the "cherry-pick" git command that gerrit generates for the patch. It will look something like: git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/88/125888/1 && git cherry-pick FETCH_HEAD
ssh deployment-puppetmaster04.deployment-prep.eqiad.wmflabs
sudo -s
cd /var/lib/git/operations/puppet
git fetch
git rebase origin/production
Paste cherry-pick command from gerrit: git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/... && git cherry-pick FETCH_HEAD
- Occasionally there will be a conflict resulting from the cherry-pick; figure out how to resolve it if that happens or git cherry-pick --abort to roll back the attempt
- You may find that you need to go back to your local working copy and rebase your patch on top of another cherry-pick that is already pending review in gerrit to cleanly resolve the conflict.
Ssh to a host that the patch should effect and force a puppet run with sudo run-puppet-agent
Log the change (i.e. go to #wikimedia-releng ^connect and use "!log").

Updating/removing a cherry-pick

When you find a bug in your proposed patch (it happens to the best of us), you'll want to remove the old cherry-picked patch and possibly add an amended one.

ssh deployment-puppetmaster04.deployment-prep.eqiad.wmflabs
sudo -s
cd /var/lib/git/operations/puppet
git rebase --interactive
Find the line for the cherry-pick you want to remove/update and delete it from the git-rebase-todo file that opens in your editor
Save the file/close your editor (:q in vim; you do use vim right?)
Cross you fingers that the rebase completes with no errors
- Clean up git with git rebase --abort if there are conflicts you can't figure out how to resolve.

If you wanted to update the cherry-pick, go back to the Cherry-picking a patch from gerrit instructions now that the old cherry-pick is removed.

FAQ

How do I get my code on the beta cluster?

To get your code deployed, you just have to have it merged in the master branch. It will eventually land on beta after a few minutes. Have a look at the Jenkins beta-code-update-eqiad job.

How does MediaWiki configuration get updated?

We use a Jenkins job that reacts on merges happening in operations/mediawiki-config.git. That triggers a Jenkins job beta-mediawiki-config-update-eqiad which connects to the bastion instance and checkout the merged revision. The job invalidates the cached configuration by touching InitialiseSettings.php.

How does the database get upgraded?

If your change is supported by MediaWiki update.php, it will be applied via the hourly Jenkins job beta-update-databases-eqiad. The job spawn a child job for each of the databases. They will simply run update.php.

How to add a new extension on beta

At first, beta is a staging area, it is not meant to be a development playground. The extension you want to add to beta must have been planned for deployment on production. If you want a dev sandbox, you should create an instance in a different labs project.

Steps to deploy a new extension on beta cluster:

Make sure you have tests registered in Jenkins / Zuul. You don't want to deploy a php linting error
Verify the extension is registered as a WMF-branched extension in make-wmf-branch (found in repos/releng/release)
Register the extension in operations/mediawiki-config.git
- Add entry point in /wmf-config/extension-list (used to generate l10n messages)
- Add a wfLoadExtension() in CommonSettings.php
Eventually create a global $wmgUse<your extension name> including the default of off/false in InitialiseSettings.php
Create settings using either CommonSettings-labs.php or InitialiseSettings-labs.php

Whenever your change to operations/mediawiki-config.git is merged, it will be deployed by Jenkins job beta-mediawiki-config-update-eqiad as described above.

Remember: you don't want the extension to be enabled if it is not ready / added to wmf branches yet.

My code introduces a feature that is not yet ready for production, should I wait to merge in master?

You definitely do not want your experiment to land on production which would potentially cause a lot of havoc. A best practice we have been following for more than a decade is to safe guard your new feature using a MediaWiki global setting. Having a setting set to false by default, would make sure the feature is not going to be enabled on production. You could then explicitly enable it on beta by editing the wmf-config/InitialiseSettings-labs.php file of operations/mediawiki-config.git.

Example highlighting how to safeguard a new feature:

$wgEnableFeatureOne = false;

function something() {
  global $wgEnableFeatureOne;
  // some code
  if ( $wgEnableFeatureOne ) {
    // Your new code
  }
  // some more code
}

Then to enable this feature on labs:

// wmf-config/InitialiseSettings-labs.php under wmfLabsSettings()
  'wgEnableFeatureOne' => [ 'default' => true ],

Get it merged. Jenkins deploys the new configuration, the feature is enabled \O/