MegaCli
MegaCli is LSI's command line interface to their MegaRAID controller family. Dell uses them in their servers as e.g. "PERC5/i" or similar. Our dataset1 server uses this RAID controller.
The tool can be downloaded from the LSI website pages on broadcom.com, e.g. here
Version 8. The ZIP file contains a ZIP file which contains two RPM files.
NOTE that this tool is generally installed on all hosts with lsi raid from a debian package now, and is called megacli (note the lower case).
Installation from the ZIP file
apt-get install unzip wget http://www.lsi.com/DistributionSystem/AssetDocument/8.00.11_Linux_MegaCLI.zip unzip 8.00.11_Linux_MegaCLI.zip MegaCliLin.zip unzip MegaCliLin.zip MegaCli-8.00.11-1.i386.rpm apt-get install rpm cd / rpm2cpio ~-/MegaCli-8.00.11-1.i386.rpm | cpio -div apt-get install libsysfs2 ln -s /opt/MegaRAID/MegaCli/MegaCli64 /usr/sbin/MegaCli
Common Parameters
Many of the commands of MegaCli make use of the following parameters:
- -aN
- Specifies the adapter. Use -a0 for the first adapter, -a1 for the second, or -aALL for all adapters.
- -PhysDrv [E:S]
- Specifies a physical drive. E is the enclosure ID, as returned by MegaCli -EncInfo -aALL. For dataset1, the possible enclosure IDs are 16 and 41. If more than one drive has to be specified, the drives are written in the form [E:S, E:S, ...].
- -Lx
- Specifies a virtual drive (aka RAID array) (where x is a number starting with zero or the string all).
Monitoring
The following commands can be used to check the status of the arrays.
Information gathering
The following script can be used to gather useful information about a failed disk for dc-ops
sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli
Battery status
megacli -AdpBbuCmd -aALL
megacli -AdpBbuCmd -GetBbuStatus -a0 | grep -e '^isSOHGood' -e '^Charger Status' -e '^Remaining Capacity' -e 'Charging'
What is the Default and the Current policy of the RAID controller depending on the BBU status
megacli -LDInfo -LAll -aAll | grep "Cache Policy:"
Force WriteBack if needed (this should only be done if you know what you are doing (e.g. miss-configuration)- normally WriteThrough is forced when there is a faulty battery/it is discharged. Checking the health/repacing the battery should be the main way to solve this, otherwise an unsafe configuration could cause data loss!):
megacli -LDSetProp -ForcedWB -Immediate -Lall -aAll
Force the BBU to relearn cycle
megacli -AdpBbuCmd -BbuLearn -aAll
Array status
To check the whole array status (ie: we have a RAID10 and we want to check the RAID status we can run)
megacli -LDInfo -L0 -a0
And that would let us know the RAID status and the information about it (and its SPANs)
Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name : RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0 Size : 1.633 TB Sector Size : 512 Mirror Data : 1.633 TB State : Optimal Strip Size : 256 KB Number Of Drives per span:2 Span Depth : 6 Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Disk's Default Encryption Type : None Bad Blocks Exist: Yes Is VD Cached: Yes Cache Cade Type : Read Only
As we can see in that RAID we have 6 SPANs and it has 2 disks per SPAN, if we are interested in checking the SPAN status (to see which of the SPANs is degraded)
megacli -CfgDsply -a0
There we can see the different SPANs, the number of disks, the status of the SPANs and the information about the disks that belong to each SPAN.
(in this case we have two as it says on PDs: 2 and those disks are Phisicaly Disk 0 and Phisical Disk 1)
SPAN: 5 Span Reference: 0x05 Number of PDs: 2 Number of VDs: 1 Number of dedicated Hotspares: 0 Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name : RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0 Size : 1.633 TB Sector Size : 512 Mirror Data : 1.633 TB State : Optimal Strip Size : 256 KB Number Of Drives per span:2 Span Depth : 6 Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Disk's Default Encryption Type : None Bad Blocks Exist: Yes Is VD Cached: Yes Cache Cade Type : Read Only Physical Disk Information: Physical Disk: 0 Enclosure Device ID: 32 Slot Number: 10 Drive's position: DiskGroup: 0, Span: 5, Arm: 0 Enclosure position: N/A Device Id: 10 WWN: 5000C50043A97C34 Sequence Number: 2 Media Error Count: 0 Other Error Count: 0 Predictive Failure Count: 0 Last Predictive Failure Event Seq Number: 0 PD Type: SAS Raw Size: 279.396 GB [0x22ecb25c Sectors] Non Coerced Size: 278.896 GB [0x22dcb25c Sectors] Coerced Size: 278.875 GB [0x22dc0000 Sectors] Sector Size: 0 Firmware state: Online, Spun Up Device Firmware Level: ES64 Shield Counter: 0 Successful diagnostics completion on : N/A SAS Address(0): 0x5000c50043a97c35 SAS Address(1): 0x0 Connected Port Number: 0(path0) Inquiry Data: SEAGATE ST3300657SS ES646SJ3NDEM FDE Capable: Not Capable FDE Enable: Disable Secured: Unsecured Locked: Unlocked Needs EKM Attention: No Foreign State: None Device Speed: 6.0Gb/s Link Speed: 6.0Gb/s Media Type: Hard Disk Device Drive Temperature :38C (100.40 F) PI Eligibility: No Drive is formatted for PI information: No PI: No PI Port-0 : Port status: Active Port's Linkspeed: 6.0Gb/s Port-1 : Port status: Active Port's Linkspeed: Unknown Drive has flagged a S.M.A.R.T alert : No Physical Disk: 1 Enclosure Device ID: 32 Slot Number: 11 Drive's position: DiskGroup: 0, Span: 5, Arm: 1 Enclosure position: N/A Device Id: 11 WWN: 5000C50054EA3600 Sequence Number: 4 Media Error Count: 0 Other Error Count: 0 Predictive Failure Count: 8 Last Predictive Failure Event Seq Number: 37843 PD Type: SAS Raw Size: 558.911 GB [0x45dd2fb0 Sectors] Non Coerced Size: 558.411 GB [0x45cd2fb0 Sectors] Coerced Size: 558.375 GB [0x45cc0000 Sectors] Sector Size: 0 Firmware state: Online, Spun Up Device Firmware Level: ES65 Shield Counter: 0 Successful diagnostics completion on : N/A SAS Address(0): 0x5000c50054ea3601 SAS Address(1): 0x0 Connected Port Number: 0(path0) Inquiry Data: SEAGATE ST3600057SS ES656SL4GP4B FDE Capable: Not Capable FDE Enable: Disable Secured: Unsecured Locked: Unlocked Needs EKM Attention: No Foreign State: None Device Speed: 6.0Gb/s Link Speed: 6.0Gb/s Media Type: Hard Disk Device Drive Temperature :40C (104.00 F) PI Eligibility: No Drive is formatted for PI information: No PI: No PI Port-0 : Port status: Active Port's Linkspeed: 6.0Gb/s Port-1 : Port status: Active Port's Linkspeed: Unknown Drive has flagged a S.M.A.R.T alert : Yes
Disk status
To check all the disks we can use:
megacli -PDList -aall
Probably, the most wanted fields we are looking at the output from this commands are:
Slot number and device ID
Firmware status (a failed disk will show Failed)
Foreign state: normally it should have "None", unless we insert a disk which has belonged to a different RAID. In that case we have to clear the previous configuration.
Smart status
Controller log output to a file
megacli -AdpEventLog -IncludeDeleted -f log.txt -aALL
Replacing Disks
Automatic array rebuild with a new disk
Normally when a disk fails, and the controller detects it, it gets marked as a bad disks (Firmware status: Failed)
It needs to be replaced, and if it is a brand new disk, as soon as it gets replaced, the controller will start to rebuild the RAID.
By using the command
megacli -PDList -aall
We can identify the disk that failed (look for the one that has Firmware status: Failed) and by looking at its Enclosure devide ID and Decide ID we can form the following command to check the % on the RAID rebuild:
megacli -PDRbld -ShowProg -PhysDrv [32:4] -aALL Rebuild Progress on Device at Enclosure 32, Slot 4 Completed 2% in 5 Minutes. Exit Code: 0x00
Once it is finished we will be able to see that the array is fine again by running:
megacli -AdpAllInfo -aALL
Device Present
================
Virtual Drives : 1
Degraded : 0
Offline : 0
Physical Devices : 14
Disks : 12
Critical Disks : 0
Failed Disks : 0
Manually array rebuild with an used disk
If we use a non new disk, which has been part of an array in the past, it will contain metadata and the controller will not start to rebuild the array until we have cleaned that configuration, as a security precaution.
First of all we need to make sure that indeed, the RAID is not rebuilding because there is a foreign configuration on a disk.
We can scan for old configurations:
megacli -CfgForeign -Scan -aALL There are 1 foreign configuration(s) on controller 0.
In that case we can verify that if it is indeed our new disk:
megacli -PDList -aall
When looking for our new disk we will realise it will have these fields:
Enclosure Device ID: 32 Device Id: 4 Firmware state: Unconfigured(good), Spun Up Foreign State: Foreign
So we know that the disk 32:4 is in a good state but it was part of an array before and needs to be cleaned up.
megacli -CfgForeign -Clear -aALL Foreign configuration 0 is cleared on controller 0. Exit Code: 0x00
Now we can check if we are good to go:
megacli -CfgForeign -Scan -aALL There is no foreign configuration on controller 0. Exit Code: 0x00
We are ready to replace the disk.
First of all we need to identify in which SPAN the old disk was and in which position.
We firstly need to go thru all the SPANs:
megacli -CfgDsply -a0 | grep "Number of Spans" Number of Spans: 6
And we need to know how many disks per SPAN we should have:
megacli -CfgDsply -a0 | grep "Number of PDs" -m1 Number of PDs: 2
There must be 2 disks per SPAN, so we need to scan all the SPANs to see which one has a disk with no information.
Normally a SPAN will show the Physical Disk information empty if a disk is missing.
In this case we stop at SPAN 2 because the information for Physical Disk 0 (in bold) is missing (compare it with the information it shows for Physical Disk 1).
And the SPAN also shows: State: Degraded
SPAN: 2 Span Reference: 0x02 Number of PDs: 2 Number of VDs: 1 Number of dedicated Hotspares: 0 Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name : RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0 Size : 1.633 TB Sector Size : 512 Mirror Data : 1.633 TB State : Degraded Strip Size : 256 KB Number Of Drives per span:2 Span Depth : 6 Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Disk's Default Encryption Type : None Bad Blocks Exist: Yes Is VD Cached: Yes Cache Cade Type : Read Only Physical Disk Information: Physical Disk: 0 Physical Disk: 1 Enclosure Device ID: 32 Slot Number: 5 Drive's position: DiskGroup: 0, Span: 2, Arm: 1 Enclosure position: N/A Device Id: 5 WWN: 5000C50043A74B14 Sequence Number: 2 Media Error Count: 12 Other Error Count: 0 Predictive Failure Count: 0 Last Predictive Failure Event Seq Number: 0 PD Type: SAS Raw Size: 279.396 GB [0x22ecb25c Sectors] Non Coerced Size: 278.896 GB [0x22dcb25c Sectors] Coerced Size: 278.875 GB [0x22dc0000 Sectors] Sector Size: 0 Firmware state: Online, Spun Up Device Firmware Level: ES64 Shield Counter: 0 Successful diagnostics completion on : N/A SAS Address(0): 0x5000c50043a74b15 SAS Address(1): 0x0 Connected Port Number: 0(path0) Inquiry Data: SEAGATE ST3300657SS ES646SJ3MN6X FDE Capable: Not Capable FDE Enable: Disable Secured: Unsecured Locked: Unlocked Needs EKM Attention: No Foreign State: None Device Speed: 6.0Gb/s Link Speed: 6.0Gb/s Media Type: Hard Disk Device Drive Temperature :38C (100.40 F) PI Eligibility: No Drive is formatted for PI information: No PI: No PI Port-0 : Port status: Active Port's Linkspeed: 6.0Gb/s Port-1 : Port status: Active Port's Linkspeed: Unknown Drive has flagged a S.M.A.R.T alert : No
So our new disk 32:4 needs to go to SPAN2 and and become Physical Disk 0 so for megacli that means row0 of the SPAN.
megacli -PdReplaceMissing -PhysDrv[32:4] -array2 -row0 -a0 Adapter: 0: Missing PD at Array 2, Row 0 is replaced.
Now we are ready to rebuild the ARRAY.
megacli -PDRbld -Start -PhysDrv[32:4] -a0 Started rebuild progress on device(Encl-32 Slot-4)
And we can check the progress
megacli -PDRbld -ShowProg -PhysDrv [32:4] -aALL Rebuild Progress on Device at Enclosure 32, Slot 4 Completed 9% in 2 Minutes.
Note that the ARRAY and the SPAN will not be set to Optimal until the disk has finished. Once it has finished.
Check and clear Preserved Cache
In cases where a disk fails and it is replaced it might happen that the controller still has cached data for said disk. Note that when in this state the controller requires manual intervention to boot and it will prompt for "preserved data cache present".
FW error description: The current operation is not allowed because the controller has data in cache for offline or missing virtual drives.
If that's the case it is possible to list all disks with preserved cache:
# megacli -GetPreservedCacheList -a0 Adapter #0 Virtual Drive(Target ID 09): Missing. Exit Code: 0x00
And clear the preserved cache for the drive in question:
# megacli -DiscardPreservedCache -L09 -a0 Adapter #0 Virtual Drive(Target ID 09): Preserved Cache Data Cleared. Exit Code: 0x00
Manually replace multiple disks
The steps to replace multiple failed disks (of different SPANs) are the same as for MegaCli#Manually array rebuild with an used disk however, we need to make sure we do one disk at the time.
Please read the MegaCli#Manually array rebuild with an used disk before continuing here, so you understand what is going to be done now:
Once we have identified the SPANs and disks that need replace, execute the Replace+Rebuild commands one by one, and do not start all of them at the same time.
Wait for each SPAN to be finished before starting with the new one.
Note: the RAID and the SPANs will not be set to Optimal until all the degraded SPANs have been fixed.
There is an example of how to operate multiple failed disks here:
https://phabricator.wikimedia.org/T149099#2751204 and this is how to start replacing each disk: https://phabricator.wikimedia.org/T149099#2762624
Replace individual disks in JBOD
Some hosts (e.g. ms-be) have a raid controller and use all disks in each own raid 0 virtual drive (essentially JBOD). It might happen that the disk shows errors to the OS and the controller thinks the disk itself is fine (i.e. doesn't mark the disk as failed). In these cases we need to fail the disk manually as follows:
# Identify the Physical Drive ID
/usr/local/lib/nagios/plugins/get-raid-status-megacli
=== RaidStatus (does not include components in optimal state)
name: Adapter #0
Virtual Drive: 16 (Target Id: 16)
RAID Level: Primary-0, Secondary-0, RAID Level Qualifier-0
State: Optimal
Number Of Drives: 1
Number of Spans: 1
Current Cache Policy: WriteBack, ReadAheadNone, Cached, No Write Cache if Bad BBU
Span: 0 - Number of PDs: 1
PD: 0 Information
Enclosure Device ID: 32
Slot Number: 14
Drive's position: DiskGroup: 16, Span: 0, Arm: 0
Media Error Count: 2360
Other Error Count: 1
Predictive Failure Count: =====> 1 <=====
Last Predictive Failure Event Seq Number: 4812
Raw Size: 7.277 TB [0x3a3812ab0 Sectors]
Firmware state: Online, Spun Up
Media Type: Hard Disk Device
Drive Temperature: 41C (105.80 F)
The disk is PD slot 14 and enclosure 32 in adapter 0, therefore:
# Mark as offline megacli -PDOffline -PhysDrv '[32:14]' -a0 # Locate the drive (blink the led) megacli -PDLocate -PhysDrv '[32:14]' -a0
These actions will trigger a task in Phabricator. After replacement the disk can be put back (and clearing the preserved cache first if needed)
megacli -GetPreservedCacheList -a0 megacli -DiscardPreservedCache -L'disk_number' -a0 megacli -CfgEachDskRaid0 WB RA Direct CachedBadBBU -a0