Mirrors

See also: Dumps/Mirror status

mirrors.wikimedia.org

The Wikimedia Foundation hosts the following public mirrors via http:

• Debian (Official)
• Ubuntu (Official)
• Tails (Official)
• Debian OpenStack Backports (Unofficial)

This service carries the dual purpose of giving back to the FOSS projects we use and as our own internal mirror.

Technical Info

1. Debian
   • Publication Method:
     • DNS A/AAAA Record for ftp.us.debian.org, Debian Code
     • Official US mirror: https://www.debian.org/mirror/sponsors
   • Sync Method: SSH Trigger, syncs are triggered by an SSH forced key command for the mirror user. Debian has StrictHostKeyChecking no set on their side, so changing our SSH host key does not affect Debian's trigger.
2. Ubuntu
   • Publication Method:
     • Official US mirror: https://launchpad.net/ubuntu/+mirror/wikimedia-archive
   • Sync Method: Systemd Timer
3. Tails
   • Publication Method:
     • DNS A Record for dl.amnesia.boum.org.
     • Mirrorbits redirector
   • Sync Method: Systemd Timer
4. Debian OpenStack Backports
   • Publication Method:
     • None, not published officially, but open to the public
   • Sync Method: Systemd Timer

Implementation Info

As of December 2021 they are hosted on mirror1001.wikimedia.org.

The server is located at the Equinix data center in Ashburn, Virginia, United States (eqiad).

The relevant Puppet class is modules/role/manifests/mirrors.pp.

The repo for the puppet code can be git cloned with "git clone https://gerrit.wikimedia.org/r/operations/puppet" from Wikimedia's Gerrit installation.

Monitoring and debugging

There are 2 Icinga checks called "Ubuntu mirror in sync with upstream" and "Debian mirror in sync with upstream" respectively.

If one of these alerts you should check on mirror1001 what happens if you manually run the check commands,

$ /usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/ubuntu
$ /usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/debian

Also you can run the actual sync command manually but make sure to run it was the user "mirror", so "$ sudo -u mirror ftpsync".

Furthermore, you can check /etc/ftpsync/ftpsync.conf to see which upstream hosts we are rsyncing from and look at /var/log/ftpsync/ to check logfiles for errors.

For Debian you can also see if you are receiving SSH triggers: tail -F /var/log/auth.log | grep forced-command

Contacting Wikimedia Maintainers

The mirror server is maintained by the Wikimedia SRE team. To report issues with mirrors.wikimedia.org, please create a ticket in Wikimedia's Phabricator and tag it with 'SRE'. You can also join the Libera.Chat IRC channels #wikimedia-operations ^connect and #wikimedia-tech ^connect.

Contacting Upstream Maintainers

1. Debian
   • Email: mirrors@debian.org, use this list to ask for changes to DNS records, may take a few days to obtain a reply
   • IRC: irc://irc.oftc.net/debian-mirrors
2. Ubuntu
   • Email: mirrors@ubuntu.com
   • IRC: ircs://libera.chat/ubuntu-mirrors
3. Tails
   • Email: tails-mirrors@boum.org, use this list to ask for changes to DNS records, may take a few days to obtain a reply
4. Debian OpenStack Backports
   • IRC: irc://irc.oftc.net#debian-openstack