Portal:Data Services/Admin/Dumps

The clouddumps servers serve dumps to clients running in Cloud VPS and the internet. See also Dumps/Dump servers.

Interfaces

NFS

A subset of Cloud VPS instances and production hosts mount NFS shares from dumps-nfs.wikimedia.org. The service IP is handled by pybal as dumps-nfs service and it is backed by one/multiple clouddumps servers, the nfsd / dumps dashboard provides an overview of the service. The implementation work and more context can be found in task T411248.

Web

Clients on the internet can download dumps via the web server, which is accessed via the dumps.wikimedia.org service name (which points to the dumps LVS service).

rsync

Mirrors have rsync access to the dumps shares. They use the dumps-rsync.wikimedia.org service name, which points to the dumps LVS service.

Failover

The web, rsync and nfs servers are load-balanced via LVS and the pool status can be managed via conftool.

Troubleshooting

If you know or suspect a problem with nfs and clouddumps make sure to perform the following to help with debug:

• Capture nfs traffic, then ideally trigger failing operations on the client side:

timeout 2m tshark -i any -w $HOSTNAME-$(date -Is)-nfs.cap 'port 2049'

• Dump clients details for later inspection:

cp -r /proc/fs/nfsd/clients $HOSTNAME-$(date -Is)-nfs-clients 2>/dev/null

In general it is safe to failover via confctl to a single server, e.g. to make clouddumps1002 the only active server:

confctl select 'service=dumps-nfs,name=clouddumps1002.wikimedia.org' set/pooled=yes
confctl select 'service=dumps-nfs,name=clouddumps1001.wikimedia.org' set/pooled=no

You will see clients migrating from one to the other after a few minutes in the nfsd dashboard

Reimaging

As of this writing, the secondary drive (which holds the dumps data) is not automatically added to /etc/fstab or mounted after a reimage.

As such, before pooling the server, you'll need to do the following:

• mkdir /srv/dumps
• blkid to get the partition UUID of the volume. Create an fstab entry using the partition ID instead of the device ID (such as /dev/sda) as the device IDs can change after a reboot.
• Verify your fstab entry is correct by running mount -a and confirming that both volumes are mounted and have the expected size and amount of data. For example:

  df -h | grep "/dev/sd"
  /dev/sdb1       438G  4.6G  411G   2% /
  /dev/sda1       196T  158T   29T  85% /srv/dumps
  

  As stated above, the device IDs may vary.
• Run Puppet. Assuming everything comes back cleanly, you should be able to repool the server. As of this writing, the dumps servers participate in 3 pools, each starting with dumps-. Check config-master for up-to-date info.