releases.wikimedia.org
releases.wikimedia.org hosts software releases (primarily packaged as tars).
It is currently served by releases2003 (standby: releases1003) (formerly: releases1001/releases2001/releases2002/releases1002) behind ATS through the DNS record 'releases.discovery.wmnet'.
Debian repository
Upload a package
On releasesX003:
cd /srv/deployment/parsoid/deploy/ dpkg-buildpackage -b deb-upload ../parsoid_0.1.12_amd64.changes
Alternatively, build locally and copy .changes and .deb to releasesX003:
scp parsoid_0.4.1_* deploy1001.eqiad:/tmp deb-upload /tmp/parsoid_0.4.1_*.changes
Check the repository: https://releases.wikimedia.org/debian/
Use the repository
Import the repository gpg key:
sudo apt-key advanced --keyserver pgp.mit.edu --recv-keys AF380A3036A03444
It should say:
gpg: key AF380A3036A03444: public key "MediaWiki releases repository <wikitech-l@lists.wikimedia.org>" imported gpg: Total number processed: 1 gpg: imported: 1
Add the repository:
sudo apt-add-repository "deb https://releases.wikimedia.org/debian jessie-mediawiki main"
Add a distro
If you need to add a new distro version, 238348 shows how 'jessie' was added in addition to trusty.
238525 shows how the default was switched.
Running reprepro commands
When running reprepro commands for general purpose ops tasks, use the "reprepro" user (e.g. sudo su -c /bin/bash reprepro) To run reprepro command you also need to run
export REPREPRO_BASE_DIR=/srv/org/wikimedia/reprepro
GPG operations
The private gpg key used by reprepro to sign distribution files lives in private.git, the public keyring lives in the public puppet repo, both are installed by puppet on the relevant machines.
Generate a new key
It might be necessary to generate a new key, it can be done by creating a temporary "gpg home", generate a new key, and replace the secure keyring in private.git, as follows:
# install -d -m 700 gpg_releases
# gpg --homedir gpg_releases --gen-key
gpg: WARNING: unsafe permissions on homedir `gpg_releases'
gpg (GnuPG) 1.4.11; Copyright (C) 2010 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
gpg: keyring `gpg_releases/secring.gpg' created
gpg: keyring `gpg_releases/pubring.gpg' created
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection? 4
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0) 150w
Key expires at Wed 12 Jun 2019 09:19:52 AM UTC
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: MediaWiki releases repository
Email address: wikitech-l@lists.wikimedia.org
Comment:
You selected this USER-ID:
"MediaWiki releases repository <wikitech-l@lists.wikimedia.org>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
pub 4096R/22250DD7 2016-07-27 [expires: 2019-06-12]
Key fingerprint = A6FD 76E2 A61C 5566 D196 D2C0 90E9 F83F 2225 0DD7
uid MediaWiki releases repository <wikitech-l@lists.wikimedia.org>
# mv gpg_releases/secring.gpg /srv/private/releases/files/secring.gpg
# cd /srv/private/releases/files/ && git commit secring.gpg
The public keyring pubring.gpg must be updated in the public puppet repo at modules/releases/files.
When updating the public keyring, import it in your local machine and send the public key to keyring servers with gpg --keyserver hkps.pool.sks-keyservers.net --send-keys KEYID.
Also remember to delete gpg_releases temporary directory once done.
Refresh signatures
It is sufficient to ask reprepro to export its files again once the new keyrings are in place (as user reprepro! see above):
cd /srv/org/wikimedia/reprepro reprepro -v export jessie-mediawiki
And make sure to purge releases.wikimedia.org urls as outlined in https://wikitech.wikimedia.org/wiki/Multicast_HTCP_purging
Failing over
Hiera data switch
In the puppet repo in hieradata/common.yaml there are 2 keys to define which of the releases* servers is currently the active one and which is the passive one. example:
releases_server: releases1003.eqiad.wmnet
releases_servers_failover:
- 'releases2003.codfw.wmnet'
Flipping these values around and then running puppet on all releases* servers is one of the main steps to fail over.
What this actually changes is which server is considered the source of rsyncing data. It changes the direction of the data sync and some firewall rules.
The complete details can be seen by uploading a change to hieradata without merging it and running it through the puppet compiler.
DNS switch
The other major step is editing the DNS record releases.discovery.wmnet to point to the other server. example to check the current state:
[releases1003:~] $ host releases.discovery.wmnet releases.discovery.wmnet is an alias for releases1003.eqiad.wmnet. releases1003.eqiad.wmnet has address 10.64.48.34 releases1003.eqiad.wmnet has IPv6 address 2620:0:861:107:10:64:48:34
This record can be found in the DNS repo in templates/wmnet under the section labeled "; misc web services with multiple backends but without geoip". One of the machine will be commented out and the other one will be active. example:
This is the big switch that changes which of the backends gets live traffic because the discovery DNS record is what is used in the configuration of the CDN (Apache traffic server) in hieradata/common/profile/trafficserver/backend.yaml.
2 services, not just 1
Note that there are 2 separate external service names configured to point to the releases backends.
Normally both are changed at the same time because both point to this single discovery DNS record. If for some reason they need to be switched separately then an edit needs to be made to the CDN backend.yaml config mentioned above.
releases 300 IN CNAME releases1003.eqiad.wmnet. ;releases 300 IN CNAME releases2003.codfw.wmnet.
Failover plans and tickets
Both of the following tickets include P85324, a plan to fail-over that was scheduled in November 2025.
task T392127 - upgrade releases hosts to bookworm
task T392127 - Releases failover process
Shell user uploads
- There are multiple shell user groups called "releasers-*" who upload releases to the backends. They need to be informed about the change and be aware to use the correct server (it will sync to the passive server as the rsync source).
They can also use the releases.discovery.wmnet DNS record to connect to but would get a warning about changed SSH host keys when the backend behind it changes. And we should not encourage just ignoring warnings like that. It's better if they use the actual host names and merely lookup the DNS record to see which host is currently behind it.
puppet/modules/admin/data$ grep releasers data.yaml releasers-mediawiki: releasers-mobile: releasers-wikidiff2: releasers-wikibase: releasers-blubber: releasers-mwcli:
Documentation updates
- https://wikitech.wikimedia.org/wiki/Releases.wikimedia.org needs to be edited to reflect the current situation (which host is the active one) or we should avoid "hardcoding" the host name in the wiki page.
- If we introduce new releases* hosts a page on Wikitech should be created from the host template which automatically links to a fingerprint page. The fingerprint page needs to be updated with the correct host key. To get the host key, SSH to the machine and run gen_fingerprints and copy/paste the result to wiki. Then, as a wiki admin, protect the page from edits of non-admins so that nobody can mess with the content. examples:
- https://wikitech.wikimedia.org/wiki/Releases1001
- https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/releases1001.eqiad.wmnet