How to perform security fixes

Make sure you're not alone on this. Make sure private patches get reviewed as Phabricator attachments, not in Gerrit, before deploying them.

This is a brief guide on how to perform security fixes. From commit draft to minor release.

The general MediaWiki documentation is:

Steps

1: Commit

Locally (on your workstation) check out the currently relevant wmf branch(es). Write the fix there and test it. Then commit to your local topic branch. You also may want to check if this commit applies cleanly to the other active wmf branch (if there is two, there is usually two, e.g. wmf/1.35-wmf.9 and wmf/1.35-wmf.10), so you're prepared to handle the merge conflict later on.

Note: Commits (especially security related ones) should, of course, always add release notes!

Prefix the commit message with "SECURITY:" to make them easy to see by others on tin.

It's also useful to make sure that a task exists in Phabricator. If one doesn't exist already, please create one. You can use the TXXXXX number for the filename. This makes tracking of patches much easier

# Create a topic branch
jdoe@laptop:~/Dev/mediawiki/core$ git checkout -b TXXXXX master

# Fix the security bug

# Add the file (or files) to the staging area
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX)$ git add path/to/file.php RELEASE-NOTES-1.35

# Confirm your stage contains what you want to do commit (no more, no less)
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX)$ git status

jdoe@laptop:~/Dev/mediawiki/core (TXXXXX)$ git commit

2: Apply on deploy host

# Convert the commit to a git patch file
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX)$ git format-patch HEAD^ --stdout > ~/TXXXXX.patch

# Copy it to your home directory on tin (scp defaults: protocol=ssh, username=local username, target directory=home directory)
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX)$ scp ~/TXXXXX.patch deploy1001.eqiad.wmnet:

# Log into tin and go to the git dir
jdoe@laptop:~/Dev/mediawiki/core$ ssh deploy1001.eqiad.wmnet
# If the patch is for an extension, go to /srv/mediawiki-staging/php-1.NN-wmf.NN/extensions/ExtensionName
jdoe@tin:~$ cd /srv/mediawiki-staging/php-1.35-wmf.10
# Sign off uses your .gitconfig (on tin) to determine name and e-mail address.
jdoe@tin:/srv/mediawiki-staging/php-1.35-wmf.10 (wmf/1.35-wmf.10)$ git am --signoff < ~/TXXXXX.patch

# Move the .patch file to /srv/patches/<branch>/core/<order>-TXXXXX.patch (for core patches)
# or /srv/patches/<branch>/extensions/ExtensionName/<order>-TXXXXX.patch (for extension patches,
# create the directory if it doesn't exist already). <order> should be chosen to be the next, sequential
# number for files in the given directory.
jdoe@tin:/srv/mediawiki-staging/php-1.35-wmf.10 (wmf/1.35-wmf.10)$ mv ~/TXXXXX.patch /srv/patches/php-1.35-wmf.10/core/01-TXXXXX.patch
jdoe@tin:~$ cd /srv/patches/
jdoe@tin:/srv/patches (master)$ git add .
# The security patches directory is a git repository of its own, to track patches.
jdoe@tin:/srv/patches (master)$ git commit -a -m 'Add php-1.35-wmf.10/core/01-TXXXXX.patch'

If there are other active branches, apply the patch there as well (see roadmap).
You may want to verify at this point that the bug is fixed on test.wikipedia.org, if possible (i.e. testing it might require adding specific permissions to your account which could expose which code is vulnerable).

3: Deploy

See also How to deploy code. If the issue is not public, make sure you set the --no-log-message to prevent it from being automatically publicly logged. When done, you should go to #wikimedia-operations ^connect and say "!log Deployed patch for Txxxxx" to log it manually without revealing potentially sensitive details such as file names.

jdoe@tin:/srv/mediawiki-staging$ scap sync-file --no-log-message php-1.35-wmf.10/includes/PathToFile.php 'Deploy security fix'

4: Request CVE

Request a CVE via https://cveform.mitre.org/ or by contacting the security team. Provide as much information as possible and your public key so as to receive an encrypted response. If you need help or guidance on creating the request, please reach out to a member of the Wikimedia Security Team.

5: Push to gerrit & make release

This step is typically done by the Security team and/or Release Engineering team on a semi-regular basis once there's a collection of security patches ready to be released.

Todo: These could be moved to a generic "Backport and do a minor release" how-to page, then put a link here.

Make release

If the vulnerability is not public, send a pre-release announcement at least 24 hours before release, announcing the availability of the update. Coordinate with Red Hat's security team to assign CVE numbers if needed.

For each patch in your release, port and commit the patch to the release branches.

jdoe@laptop:~/Dev/mediawiki/core$ git checkout -t gerrit/REL1_19
# Check to make sure the patch applys, if not you'll need to backport
jdoe@laptop:~/Dev/mediawiki/core (REL1_19)$ git apply --check < ~/TXXXXX.patch
jdoe@laptop:~/Dev/mediawiki/core (REL1_19)$ git am ~/TXXXXX.patch

Update and commit RELEASE-NOTES and DefaultSettings.php version number

Tag the release

jdoe@laptop:~/Dev/mediawiki/core (REL1_19)$ git tag -a '1.19.6' -m "Security Release 1.19.6"

Copy the current version of the release branch'es extensions into extensions/ TODO: This is painful and needs to be done better

Get make-release from mediawiki/tools/release.git. Run make-release:

jdoe@laptop:~/Dev/mediawiki/tools/make-release$ ./make-release --git-root ~/Dev/mediawiki/core/ 1.19.6
Was 1.19.5 the previous release? [y/n] y
 Cloning core...

Decrypt your private key when prompted to sign tarballs. Keep the output from make-release for the release announcement email.

Copy the build branch to tin

jdoe@laptop:~/Dev/mediawiki/tools/make-release$ cd build/
jdoe@laptop:~/Dev/mediawiki/tools/make-release/build$ scp -r 1.19 deploy1001.eqiad.wmnet:.

Have someone with root move the files to download.wikimedia.org (dataset2:/data/xmldatadumps/public/mediawiki). Warning: Once the files are copied, they are publicly available, since the directory index listings are available on the server.

Send release announcement emails.

Push to gerrit

# Commit to the wmf branch in gerrit. Especially important because the clone on tin is
# now 1 commit ahead of the gerrit repository
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX)$ git push gerrit HEAD:refs/for/wmf/1.35-wmf.10
# Creates new gerrit change, self-merge this one through the web interface
# Okay, since it is already deployed anyway.

# Now for the release. First to master.
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX)$ git push gerrit HEAD:refs/for/master

# Make note of the git commit hash 
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX)$ git show HEAD --stat

# Then to the other release branch(es).
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX)$ git checkout -b TXXXXX/REL1_19 gerrit/REL1_19
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX/REL1_19)$ git cherry-pick 9023aa2d830029da5745e92212f03ddbc71da4c2
# Make sure to update the release notes properly (probably causes a conflict, either way update the right file)
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX/REL1_19)$ git add RELEASE-NOTES-1.19
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX/REL1_19)$ git commit --amend
jdoe@laptop:~/Dev/mediawiki/core (TXXXXX/REL1_19)$ git push gerrit HEAD:refs/for/REL1_19

# Creates a new gerrit change for each push. Merge as appropriate.