Portal:Data Services/Admin/Creating a DRBD set

From Wikitech
This page contains historical information. It may be outdated or unreliable.

This is a quick guide to how you can create a quick test NFS cluster with DRBD in the cloudstore project. You'd mostly do this for practice and testing.

Create a pair of VMs

They don't need to be particularly big if you are just doing this to try things out. However, if they are named with a prefix of cloudstore-dev-, then you get the role role::wmcs::nfs::test applied, which will set up all the package installs you will need. However, if there is an existing cluster configured in the Horizon puppet prefix, your new instances will probably be broken. I suggest creating your own prefix, like cloudstore-test- or something like that, using the material in the cloudstore-dev prefix as a template. There is a project local puppetmaster for testing patches, though I'm not sure it is applied to the project by default right now.

Create and attach a cinder vol for each one

You don't need to run the /usr/local/sbin/wmcs-prepare-cinder-volume script since you are going to carve this up with LVM. See Help:Adding_Disk_Space_to_Cloud_VPS_instances. On a hardware server, you'd likely be using a hardware-defined RAID 10 volume for this purpose. If you have successfully attached the cinder vol, you should see it (most likely as /dev/sdb) when you run lsblk.

Use LVM

LVM can use paritions or disks of all kinds. In this case you just need to feed it the cinder volume. All commands will need root or sudo and need to be run on both VMs, separately.

  • pvcreate /dev/sdb
  • vgcreate vd /dev/sdb Note: you can name the volume group whatever you like, but vd is what our old LVM puppet module used. The hardware servers are using other names like "data".
  • vgs This is just to admire your handywork.
  • From here, the volumes you create must match the DRBD definitions you are using in puppet. If you are using the current version of role::wmcs::nfs::test then it will need three volumes (of whatever size): test, tools, misc (similar to what is on the primary cluster). The important thing is that the volumes are named according to what puppet expects and the sizes are identical between VMs. So assuming you have a cinder vol larger than 11 GB on both VMs:
    • lvcreate -L 5G -n tools-project vd
    • lvcreate -L 5G -n misc-project vd
    • lvcreate -L 1G -n test vd

Now you can run lvs and see your logical volumes waiting to be made into DRBD vols.

Claim the volumes for DRBD

DRBD need to create metadata on the volumes. If there is a filesystem on them, you'll need to destroy that in a small way at least in order to get the commands to let you do this. Creating DRBD metadata is a destructive operation, and you should never force it on a disk that you have data that you must not lose on.

The following commands are dependent on the DRBD configuration on the server or they would not work like this. Puppet should set that up for you with the drbd resource that is defined there, if you used a valid role. Continuing as though you are using the role::wmcs::nfs::test role:

  • drbdadm create-md test -- The final part of this command is the volume name, which you'll find in the DRBD config.
  • drbdadm create-md tools
  • drbdadm create-md misc
  • systemctl start drbd -- This is actually an /etc/init.d script that loads the modules DRBD uses and runs a bunch of things including "drbdadm up all" effectively.

Check the status

Now the script sudo drbd-overview will show something more interesting. Generally both sides will initially be marked as "Secondary", which isn't very useful. To start serving NFS, one side must become the primary and NFS must be started. You will want to have defined the expected active server in the hiera values in your Horizon setup. Again, modify this for whatever volumes you are using if you aren't using the existing test role in puppet.

Bring up a DRBD primary and NFS

Whichever server that is, only on that server run:

  • drbdadm primary all --force -- On a fresh cluster, you typically will need to force the action. This overrides some checks for "up-to-date" data. There's no data in the beginning so logically it cannot be up-to-date yet.
  • mount -o noatime /dev/drbd4 /srv/tools
  • mount -o noatime /dev/drbd1 /srv/test
  • mount -o noatime /dev/drbd3 /srv/misc
  • systemctl start nfs-server -- Once this is run, nfs-exportd might stop failing...if it doesn't run on your server (I don't remember if it runs for the test role), then you can make an /etc/exports file to try out mounting and things on a client. Things written to the disks should be replicated over to the replica.

Check again!

Now sudo drbd-overview should look like a properly replicating cluster. An example is: 

root@cloudstore-dev-01:~# drbd-overview
 1:test/0   Connected Primary/Secondary UpToDate/UpToDate /srv/test  ext4 992M 2.6M 923M 1%
 3:misc/0   Connected Primary/Secondary UpToDate/UpToDate /srv/misc  ext4 4.9G 21M  4.6G 1%
 4:tools/0  Connected Primary/Secondary UpToDate/UpToDate /srv/tools ext4 4.9G 21M  4.6G 1%
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Portal:Data_Services/Admin/Creating_a_DRBD_set&oldid=2148412"