Wikimedia Cloud Services team/EnhancementProposals/SpareDisks
Problem Statement
WMCS servers currently do not have hot spares disks added to their RAID configuration.
We depend on Dell's 24-hour support contract to deliver new replacements to us.
Since we standardize on RAID-10, a single disk failure means that the whole volume has no redundancy anymore and it's at the mercy of the remaining disk in the pair that failed to keep working until a replacement disk arrives, is actually installed and completes the rebuilding process. This means we could be running without redundancy for a few days.
Due to the way that RAID-10 and SSDs work, wear levels could be the same for both SSDs in a pair. So both are at high risk of failure, meaning the remaining disk could fail at any moment too.
Additionally, miscommunication with Dell support or shipping delays could mean additional delays. We also don't have 24x7 DC-Ops staff to start working on a disk replacement immediately and there are organizational challenges in monitoring/alerting that could add even more delays.
Proposal
Adopt a new RAID standard where 2 disks are defined as hot spares. They will become active immediately after a failure is detected by the RAID controller, giving all teams more time to react and reducing the time window where we are without redundancy in our RAID volumes.
In essence, trade storage capacity for faster MTTR.
Technical Impact
Here is the current situation for our various servers:
| Server | Vendor | Disks | Type | Raw Capacity | Current RAID-10 | RAID-10 w/ spares | Variation | Current Disk Usage |
|---|---|---|---|---|---|---|---|---|
| labvirt1001 | HP | 16 | 15k 300GB | 4.8TB | 2.4TB | 2.1TB | -12.5% | 1.2TB |
| labvirt1002 | HP | 16 | 15k 300GB | 4.8TB | 2.4TB | 2.1TB | -12.5% | 1.2TB |
| labvirt1003 | HP | 16 | 15k 300GB | 4.8TB | 2.4TB | 2.1TB | -12.5% | 0.9TB |
| labvirt1004 | HP | 16 | 15k 300GB | 4.8TB | 2.4TB | 2.1TB | -12.5% | 1.2TB |
| labvirt1005 | HP | 16 | 15k 300GB | 4.8TB | 2.4TB | 2.1TB | -12.5% | 0.9TB |
| labvirt1006 | HP | 16 | 15k 300GB | 4.8TB | 2.4TB | 2.1TB | -12.5% | 0.7TB |
| labvirt1007 | HP | 16 | 15k 300GB | 4.8TB | 2.4TB | 2.1TB | -12.5% | 1.3TB |
| labvirt1008 | HP | 16 | 15k 300GB | 4.8TB | 2.4TB | 2.1TB | -12.5% | 0.5TB |
| cloudvirt1009 | HP | 16 | 15k 300GB | 4.8TB | 2.4TB | 2.1TB | -12.5% | - |
| cloudvirt1012 | HP | 6 | SSD 1.6TB | 9.6TB | 4.8TB | 3.2TB | -33% | - |
| cloudvirt1013 | HP | 6 | SSD 1.6TB | 9.6TB | 4.8TB | 3.2TB | -33% | 0.3TB |
| cloudvirt1014 | HP | 6 | SSD 1.6TB | 9.6TB | 4.8TB | 3.2TB | -33% | 0.5TB |
| cloudvirt1015 | Dell | 10 | SSD 1.6TB | 16TB | 8TB | 6.4TB | -20% | - |
| cloudvirt1016 | Dell | 10 | SSD 1.6TB | 16TB | 8TB | 6.4TB | -20% | 2.1TB |
| cloudvirt1017 | Dell | 10 | SSD 1.6TB | 16TB | 8TB | 6.4TB | -20% | 1.5TB |
| cloudvirt1018 | Dell | 10 | SSD 1.6TB | 16TB | 8TB | 6.4TB | -20% | 1.1TB |
| cloudvirt1019 | HP | 10 | SSD 1.6TB | 16TB | 8TB | 6.4TB | -20% | 1.5TB |
| cloudvirt1020 | HP | 10 | SSD 1.6TB | 16TB | 8TB | 6.4TB | -20% | - |
| cloudvirt1021 | Dell | 10 | SSD 1.6TB | 16TB | 8TB | 6.4TB | -20% | 1.3TB |
| cloudvirt1022 | Dell | 10 | SSD 1.6TB | 16TB | 8TB | 6.4TB | -20% | 2.8TB |
| cloudvirt1023 | Dell | 10 | SSD 1.6TB | 16TB | 8TB | 6.4TB | -20% | 2.4TB |
| cloudvirt1024 | Dell | 10 | SSD 1.6TB | 16TB | 8TB | 6.4TB | -20% | 0.2TB |
| cloudvirt1025 | Dell | 6 | SSD 1.8TB | 10.8TB | 5.4TB | 3.6TB | -33% | 0.5TB |
| cloudvirt1026 | Dell | 6 | SSD 1.8TB | 10.8TB | 5.4TB | 3.6TB | -33% | 1.1TB |
| cloudvirt1027 | Dell | 6 | SSD 1.8TB | 10.8TB | 5.4TB | 3.6TB | -33% | 1TB |
| cloudvirt1028 | Dell | 6 | SSD 1.8TB | 10.8TB | 5.4TB | 3.6TB | -33% | 1TB |
| cloudvirt1029 | Dell | 6 | SSD 1.8TB | 10.8TB | 5.4TB | 3.6TB | -33% | 0.5TB |
| cloudvirt1030 | Dell | 6 | SSD 1.8TB | 10.8TB | 5.4TB | 3.6TB | -33% | 1.5TB |
| cloudvirtan1001 | Dell | 12 | 7.2k 4TB | 48TB | 24TB | 20TB | -17% | - |
| cloudvirtan1002 | Dell | 12 | 7.2k 4TB | 48TB | 24TB | 20TB | -17% | - |
| cloudvirtan1003 | Dell | 12 | 7.2k 4TB | 48TB | 24TB | 20TB | -17% | - |
| cloudvirtan1004 | Dell | 12 | 7.2k 4TB | 48TB | 24TB | 20TB | -17% | - |
| cloudvirtan1005 | Dell | 12 | 7.2k 4TB | 48TB | 24TB | 20TB | -17% | - |
| labstore1004 | Dell | 26 | 7.2k 2TB | 52TB | 26TB | 24TB | -8% | - |
| labstore1005 | Dell | 26 | 7.2k 2TB | 52TB | 26TB | 24TB | -8% | - |
| labstore1006 | HP | 12 | 7.2k 6TB | 72TB | 36TB | 30TB | -17% | - |
| labstore1007 | HP | 12 | 7.2k 6TB | 72TB | 36TB | 30TB | -17% | - |
| cloudstore1008 | Dell | 12 | 7.2k 6TB | 72TB | 36TB | 30TB | -17% | - |
| cloudstore1009 | Dell | 12 | 7.2k 6TB | 72TB | 36TB | 30TB | -17% | - |
Important notes:
- Nominal disk capacity used (base 10 sizes published by vendor)
- Disks dedicated to operating system are ignored
- cloudvirtan* are "owned" by the Analytics team and may or may not be sized to allow for spares
- labstore100{4,5,6,7} are scheduled to be decommissioned with instances on cloudvirt10{19,20} as the replacement
Timeline
If this proposal is accepted, we could adopt two strategies for deployment:
- Piggyback on our migration to Stretch/Mitaka on the hypervisors and take each reimage opportunity to modify the RAID configuration
- Schedule downtime for the remaining hypervisors that are already in Stretch/Mitaka so they're drained, reimaged and put back into production
Since draining hypervisors is a very disruptive process, the complete implementation of this proposal would have to take into account how much downtime we are comfortable with.
It's expected to be a year goal if not more.
Voting
Please add more stakeholders as needed. Vote Yes/No and a justification.
| Name | Vote | Comment |
|---|---|---|
| Andrew Bogott | - | - |
| Arturo Borrero | Yes | We should increase robustness and resilience of CloudVPS. I know this involves capacity/budget/refresh planning. |
| Brooke Storm | - | - |
| Bryan Davis | Yes | Support for the piggyback strategy + investigating "converged infrastructure" idea of re-purposing cloudvirt local storage as Ceph storage that is exposed back to the cloudvirts for instance storage. |
| Giovanni Tirloni | Yes | Reasons: Engineer time is more expensive than cost of spare disks. Lack of redundancy is unacceptable in face of data loss (which has already occurred). We cannot maintain any meaningful SLA with humans in the critical path. |
Decision
While some team members didn't cast their vote formally in this document, the majority seemed to agree that adopting hot spare disks was a good strategy in a meeting held on Feb 12.
We will:
- Reconfigure RAID arrays to add hot spares at every opportunity we have (reimages, moving hypervisors to the new eqiad1 region, etc)
- We will not drain existing hypervisors simply to reconfigure RAID because it's a too time-consuming process
- New servers being bought for codfw and only serving dev/test purposes will have only 1 hot spare disk instead of 2 (like in production hypervisors)
- Investigate using the hypervisors as Ceph nodes themselves in a "converged infrastructure" approach
RAID configuration
Initial RAID configuration is done using the last two disks as hot spares. As disks fail and get replaced the hot spares will end up being in different slots though.
Dell servers
Reboot and reconfigure it through the UI interface.
Remember to go into 'Advanced' while creating the volume and select 'Add hot spares' and 'Initialize'.
When selecting the disks for the volume, leave the last 2 disks unselected. Unintuitively, a new window will pop up after the volume is creating asking you to select the spare disks.
HP servers (Gen8)
There is an issue pressing F3 to delete existing RAID volume. If possible, run commands from Linux.
Show current status:
=> ctrl slot=0 pd all show
Smart Array P420i in Slot 0 (Embedded)
array A
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)
array B
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 300 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 300 GB, OK)
physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 300 GB, OK)
physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 300 GB, OK)
physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 300 GB, OK)
physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 300 GB, OK)
physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 300 GB, OK)
physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 300 GB, OK)
physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 300 GB, OK)
physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 300 GB, OK)
physicaldrive 1I:1:13 (port 1I:box 1:bay 13, SAS, 300 GB, OK)
physicaldrive 2I:1:14 (port 2I:box 1:bay 14, SAS, 300 GB, OK)
physicaldrive 2I:1:15 (port 2I:box 1:bay 15, SAS, 300 GB, OK)
physicaldrive 2I:1:16 (port 2I:box 1:bay 16, SAS, 300 GB, OK)
physicaldrive 2I:1:17 (port 2I:box 1:bay 17, SAS, 300 GB, OK)
physicaldrive 2I:1:18 (port 2I:box 1:bay 18, SAS, 300 GB, OK)
Delete existing RAID volume:
=> ctrl slot=0 ld 2 delete forced
Warning: Deleting an array can cause other array letters to become renamed.
E.g. Deleting array A from arrays A,B,C will result in two remaining
arrays A,B ... not B,C
Create new RAID volume (leaving 2 disks for spares):
=> ctrl slot=0 create type=ld raid=1+0 drives=1I:1:3,1I:1:4,1I:1:5,1I:1:6,1I:1:7,1I:1:8,1I:1:9,1I:1:10,1I:1:11,1I:1:12,1I:1:13,2I:1:14,2I:1:15,2I:1:16
Add hot spares to the array:
=> ctrl slot=0 array B add spares=2I:1:17,2I:1:18
Show RAID volume:
=> ctrl slot=0 ld 2 show
Smart Array P420i in Slot 0 (Embedded)
array B
Logical Drive: 2
Size: 1.9 TB
Fault Tolerance: 1+0
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 256 KB
Full Stripe Size: 1792 KB
Status: OK
Caching: Enabled
Unique Identifier: 600508B1001C2FD858B7C664FB32BECD
Disk Name: /dev/sdb
Mount Points: None
Logical Drive Label: 045EEA760014380311954D026CC
Mirror Group 1:
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 300 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 300 GB, OK)
physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 300 GB, OK)
physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 300 GB, OK)
physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 300 GB, OK)
physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 300 GB, OK)
physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 300 GB, OK)
Mirror Group 2:
physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 300 GB, OK)
physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 300 GB, OK)
physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 300 GB, OK)
physicaldrive 1I:1:13 (port 1I:box 1:bay 13, SAS, 300 GB, OK)
physicaldrive 2I:1:14 (port 2I:box 1:bay 14, SAS, 300 GB, OK)
physicaldrive 2I:1:15 (port 2I:box 1:bay 15, SAS, 300 GB, OK)
physicaldrive 2I:1:16 (port 2I:box 1:bay 16, SAS, 300 GB, OK)
Drive Type: Data
LD Acceleration Method: Controller Cache
Show all disks (confirm there are 2 spares):
=> ctrl slot=0 pd all show
Smart Array P420i in Slot 0 (Embedded)
array A
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)
array B
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 300 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 300 GB, OK)
physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 300 GB, OK)
physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 300 GB, OK)
physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 300 GB, OK)
physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 300 GB, OK)
physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 300 GB, OK)
physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 300 GB, OK)
physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 300 GB, OK)
physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 300 GB, OK)
physicaldrive 1I:1:13 (port 1I:box 1:bay 13, SAS, 300 GB, OK)
physicaldrive 2I:1:14 (port 2I:box 1:bay 14, SAS, 300 GB, OK)
physicaldrive 2I:1:15 (port 2I:box 1:bay 15, SAS, 300 GB, OK)
physicaldrive 2I:1:16 (port 2I:box 1:bay 16, SAS, 300 GB, OK)
physicaldrive 2I:1:17 (port 2I:box 1:bay 17, SAS, 300 GB, OK, spare)
physicaldrive 2I:1:18 (port 2I:box 1:bay 18, SAS, 300 GB, OK, spare)
HP servers (Gen9)
Enter the HP RAID configuration:
- Reboot server
- Press ESC+9 to enter menu
- Select RAID controler on slot 1
- Select option to open configuration utility
- Wait for "error: no such device: HPEZCD260" message to disappear
Verify current situation:
=> controller slot=1 ld 1 show
Smart Array P840 in Slot 1
Array A
Logical Drive: 1
Size: 7.3 TB
Fault Tolerance: 1+0
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 256 KB
Full Stripe Size: 1280 KB
Status: OK
MultiDomain Status: OK
Caching: Disabled
Unique Identifier: 600508B1001CB0D3B3EFD3A1715AB007
Disk Name: /dev/sdd
Mount Points: None
Logical Drive Label: 0110F6E7PDNNF0ARH8015T82C5
Mirror Group 1:
physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA SSD, 1.6 TB, OK)
physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA SSD, 1.6 TB, OK)
physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA SSD, 1.6 TB, OK)
physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA SSD, 1.6 TB, OK)
physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA SSD, 1.6 TB, OK)
Mirror Group 2:
physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA SSD, 1.6 TB, OK)
physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA SSD, 1.6 TB, OK)
physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SATA SSD, 1.6 TB, OK)
physicaldrive 2I:2:1 (port 2I:box 2:bay 1, SATA SSD, 1.6 TB, OK)
physicaldrive 2I:2:2 (port 2I:box 2:bay 2, SATA SSD, 1.6 TB, OK)
Drive Type: Data
LD Acceleration Method: SSD Smart Path
Delete existing array:
=> ctrl slot=1 array A delete forced
Warning: Deleting the specified device(s) will result in data being lost.
Continue? (y/n) y
Confirm all disks are now unassigned:
=> ctrl slot=1 pd all show
Smart Array P840 in Slot 1
Unassigned
physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA SSD, 1.6 TB, OK)
physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA SSD, 1.6 TB, OK)
physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA SSD, 1.6 TB, OK)
physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA SSD, 1.6 TB, OK)
physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA SSD, 1.6 TB, OK)
physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA SSD, 1.6 TB, OK)
physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA SSD, 1.6 TB, OK)
physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SATA SSD, 1.6 TB, OK)
physicaldrive 2I:2:1 (port 2I:box 2:bay 1, SATA SSD, 1.6 TB, OK)
physicaldrive 2I:2:2 (port 2I:box 2:bay 2, SATA SSD, 1.6 TB, OK)
Create a new array / logic device (leaving two disks to be spares later):
=> ctrl slot=1 create type=ld drives=1I:1:5,1I:1:6,1I:1:7,1I:1:8,2I:1:1,2I:1:2,2I:1:3,2I:1:4 raid=1+0 forced
Warning: SSD Over Provisioning Optimization will be performed on the physical
drives in this array. This process may take a long time and cause this
application to appear unresponsive. Continue? (y/n)y
Add the last 2 disks as spares:
=> ctrl slot=1 array all add spares=2I:2:1,2I:2:2
Resources
Add papers, external links, etc, that support proposal