Dumps/Dumpsdata hosts
XML Dumpsdata hosts
Hardware
We have various hosts:
- Dumpsdata1001 in eqiad, production xml/sql dumps nfs spare:
- Hardware/OS: PowerEdge R730xd, Debian 8 (buster), 32GB RAM, 1 quad-core Xeon ES2623 cpu, HT enabled
- Disks: 12 4TB disks in 1 12-disk raid10 volume; two 1T disks in raid 1 for the OS
- Dumpsdata1002 in eqiad, production misc dumps nfs:
- Hardware/OS: PowerEdge R730xd, Debian 10 (buster), 32GB RAM, 1 quad-core Xeon ES2623 cpu, HT enabled
- Disks: 12 4TB disks in 1 12-disk raid10 volume; two 1T disks in raid 1 for the OS
- Dumpsdata1003 in eqiad, production xml/sql dumps nfs spare:
- Hardware/OS: PowerEdge R730xd, Debian 10 (buster), 64GB RAM, 1 quad-core Xeon Silver 4112 cpu, HT enabled
- Disks: 12 4TB disks in 1 12-disk raid10 volume; two 1T disks in raid 1 for the OS
- Dumpsdata1004 in eqiad, production xml/sql dumps nfs fallback
- Hardware: coming soon
- Disks: coming soon
- Dumpsdata1005 in eqiad, production xml/sql dumps nfs primary
- Hardware: coming soon
- Disks: coming soon
Services
The production host is nfs-mounted on the snapshot hosts; generated dumps are written there and rsynced from there to the web and rsync servers.
Deploying a new host
You'll need to set up the raid arrays by hand. The single LVM volume mounted on /data is ext4.
You'll want to make sure your host has the right partman recipe in netboot.cfg, at this writing dumpsdata100X.cfg. This will set up sda1 as /boot, one lvm as / and one as /data.
Install in the usual way (add to puppet, copying a pre-existing production dumpsdata host stanza, set up everything for PXE boot and go). You will need to add the new host to the dumps_web_rsync_server_clients[ipv4][internal] list in common.yaml. You will also need to add an entry for it in hieradata/hosts; copy either the stanza for the primary generator host, if the new host is to replace it, or the stanza for a secondary (fallback) host, if the new host is to become a fallback.
Additionally you must add your host to $peer_hosts
in profile::dumps::rsyncer
and profile::dumps::rsyncer_peer
so that rsyncs can be done from this host to the rest of the dumpsdata and dumps web server hosts. This must be done for both primary and secondary hosts.
If it is a fallback host, data must be rsynced to it from the primary on a regular basis. To make this happen, add the hostname to the xmlremotedirs
and miscremotedirs
arguments as passed from profile::dumps::generation::server::primary
; you'll need to know the rsync path to the public directory for the dumps on the new host, in case it is set up differently than the rest, and you'll likewise need to know the rsync path to the directory for misc ('other') dumps. Because the initial rsync can be quite timeconsuming, it's best to do a manual rsync first from one of the web servers, and then enable the rolling rsync in puppet.
If it is to be a primary host and the old primary is to go away, when you are ready to make the switch you will need to change profile::dumps::generation::worker::common
so that the dumpsdatamount
resource mounts the new server's filesystem on the snapshot hosts instead of mounting the old primary server.
Reimaging an old host
This assumes you are following the Server Lifecycle/Reimage procedure.
You likely want to preserve all the data on the /data
filesystem. To make this happen, in netboot.cfg you'll want to set your host to use dumpsdata100X-no-data-format.cfg. This requires manual intervention during partitioning. Sepcifically, you'll need to select:
- /dev/sda1 to use as ext4, mount at
/boot
, format - data vg to use as ext4, mount at
/data
, preserve all contents - root vg to use as ext4, mount at
/
, format
Write the changes to disk and the install will carry on as desired.
Space issues
These hosts should have three sets of xml/sql dumps at all times, thus ensuring that there is always one set that contains full revision history, usable for prefetch for the next such run. If we start getting low on space, this is a huge issue; we need to head that off well before it happens.