Dumps/Mirror status

See also: Mirrors

Current mirrors

We keep track of current mirrors for dumps in the operations/puppet repo (code) in hiera; check common/profile/dumps/distribution.yaml for the entry profile::dumps::distribution::mirrors. Adding the host information here will add the mirror to the list of hosts which may rsync from our dumps servers.

Mirrors that are active should be marked as such; mirrors that are not yet up or which seem to have lapsed are marked as not active. The rest of the fields should be self-explanatory.

Adding a mirror

Requirements: Before you add a mirror, discuss bandwidth and storage with the contact person. Current storage requirements are listed at mirroring. Note that dumps are run twice a month and so bandwidth includes the mirror pulling the latest copy of all dumps on all projects of the types of dumps they choose to mirror.

Phab: As soon as an organization has agreed to host some dumps or datasets, add a Phabricator task for it tagged with the Dumps-generation project.

Config: Once a decision has been reached on what to be mirrored, you can add the relevant information as described under 'Current Mirrors', above.

Mirroring: The host itself can rsync either from another mirror, if they are set up for rsync, or from us (dumps-rsync.wikimedia.org). We have strict bandwidth caps but we provide rsync stanzas for pulling copies of just the number of xml/sql dumps desired; see the puppet template for the rsync config.

If an organization wishes to pull from another mirror using this convenient setup, they can pick up the appropriate rsync-inc-last-n.txt file (example) from the mirror at the top level, and put together an appropriate rsync command on their end.

Announcing: Once this is done and the mirror is up to date, you should add it to the list of current mirrors on Meta and email an announcement to xmldatadumps-l.

Boooooo: Note that at this time there is no automatic redirection of dumps downloaders to our mirror sites.

In progress mirrors

See Phabricator for mirrors being set up/worked on.