Dumps/Mirror status

See also: Mirrors

Current mirrors

We keep track of current mirrors for dumps in the operations/puppet repo (code) in hiera; check common/profile/dumps/distribution.yaml for the entry profile::dumps::distribution::mirrors. Adding the host information here (ipv4, ipv6) will add the mirror to the list of hosts which may rsync from our dumps servers.

Mirrors that are active should be marked as such; mirrors that are not yet up or which seem to have lapsed are marked as not active. The rest of the fields should be self-explanatory.

Adding a mirror

Requirements: Before you add a mirror, discuss bandwidth and storage with the contact person. Current storage requirements are listed at mirroring. Note that dumps are run twice a month and so bandwidth includes the mirror pulling the latest copy of all dumps on all projects of the types of dumps they choose to mirror.

Phab: As soon as an organization has agreed to host some dumps or datasets, add a Phabricator task for it tagged with the Dumps-generation project.

Config: Once a decision has been reached on what to be mirrored, you can add the relevant information as described under 'Current Mirrors', above.

Mirroring: The host itself can rsync either from another mirror, if they are set up for rsync, or from us. We have strict bandwidth caps but we provide rsync stanzas for pulling copies of just the number of xml/sql dumps desired; see the puppet template for the rsync config.

If an organization wishes to pull from another mirror using this convenient setup, they can pick up the appropriate rsync-inc-last-n.txt file (example) from the mirror at the top level, and put together an appropriate rsync command on their end.

Announcing: Once this is done and the mirror is up to date, you should add it to the list of current mirrors on Meta, email an announcement to xmldatadumps-l, and add it to the mirrors web page in our puppet repo.

Boooooo: Note that at this time there is no automatic redirection of dumps downloaders to our mirror sites.

In progress mirrors

See Phabricator for mirrors being set up/worked on.