SRE/Dc-operations/Sw raid rebuild directions

SRE Data Center Operations

DC Operations | About | Projects & Workboards | IRC: #wikimedia-dcops ^connect
HW Troubleshooting | HW Specific Documentation

When a defective disk is swapped out on a sw raid, it is not automatically rebuilt. Rebuilding requires adding the new disk in with the following procedure (uses wdqs2007 disk replacement from this task as an example):

check to see if the new disk is detected along with the existing disks:

sudo lshw -class disk

copy the parition table of sda to sdh (sdh was replaced)

sudo sgdisk -R /dev/sdh /dev/sda

create a random guid for sdh

sudo sgdisk -G /dev/sdh

audit output of both disks to ensure they now match

sudo sgdisk -p /dev/sda

sudo sgdisk -p /dev/sdh

add the new SSD back into the array

sudo mdadm --manage /dev/md0 --add /dev/sdh2

In some cases, mdraid may automatically create and add the new disk to a separate sw raid. This will manifest with the above command returning

mdadm: Cannot open /dev/sdh2: Device or resource busy

as well as /proc/mdstat showing an inactive raid that isn't md0, for instance

cat /proc/mdstat 
Personalities : [raid1] [linear]  [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md127 : inactive sda2[7](S)
     937267200 blocks super 1.2
      
md0 : active raid1 sdb2[1]
     937267200 blocks super 1.2 [2/1] [_U]
     bitmap: 6/7 pages [24KB], 65536KB chunk

In that case, copy the inactive raid id and stop it

sudo mdadm --manage /dev/md127 --stop then retry adding the device to /dev/md0

check the status

robh@wdqs2007:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid10 sdh2[8] sda2[0] sde2[4] sdf2[5] sdg2[6] sdc2[2] sdd2[3] sdb2[1]
3749068800 blocks super 1.2 512K chunks 2 near-copies [8/7] [UUUUUUU_]
[>....................]  recovery =  0.0% (132736/937267200) finish=470.6min speed=33184K/sec
bitmap: 28/28 pages [112KB], 65536KB chunk
unused devices: <none>