Jump to content

MegaCli

From Wikitech

MegaCli is LSI's command line interface to their MegaRAID controller family. Dell uses them in their servers as e.g. "PERC5/i" or similar. Our dataset1 server uses this RAID controller.

The tool can be downloaded from the LSI website pages on broadcom.com, e.g. here Version 8. The ZIP file contains a ZIP file which contains two RPM files.

NOTE that this tool is generally installed on all hosts with lsi raid from a debian package now, and is called megacli (note the lower case).

Installation from the ZIP file

apt-get install unzip
wget http://www.lsi.com/DistributionSystem/AssetDocument/8.00.11_Linux_MegaCLI.zip 
unzip  8.00.11_Linux_MegaCLI.zip  MegaCliLin.zip
unzip  MegaCliLin.zip MegaCli-8.00.11-1.i386.rpm

apt-get install rpm
cd /
rpm2cpio ~-/MegaCli-8.00.11-1.i386.rpm | cpio -div
apt-get install libsysfs2
ln -s /opt/MegaRAID/MegaCli/MegaCli64 /usr/sbin/MegaCli

Common Parameters

Many of the commands of MegaCli make use of the following parameters:

-aN
Specifies the adapter. Use -a0 for the first adapter, -a1 for the second, or -aALL for all adapters.
-PhysDrv [E:S]
Specifies a physical drive. E is the enclosure ID, as returned by MegaCli -EncInfo -aALL. For dataset1, the possible enclosure IDs are 16 and 41. If more than one drive has to be specified, the drives are written in the form [E:S, E:S, ...].
-Lx
Specifies a virtual drive (aka RAID array) (where x is a number starting with zero or the string all).

Monitoring

The following commands can be used to check the status of the arrays.

Information gathering

The following script can be used to gather useful information about a failed disk for dc-ops

sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli

Battery status

megacli -AdpBbuCmd -aALL
megacli -AdpBbuCmd -GetBbuStatus -a0 | grep -e '^isSOHGood' -e '^Charger Status' -e '^Remaining Capacity' -e 'Charging'

What is the Default and the Current policy of the RAID controller depending on the BBU status

megacli -LDInfo -LAll -aAll | grep "Cache Policy:"

Force WriteBack if needed (this should only be done if you know what you are doing (e.g. miss-configuration)- normally WriteThrough is forced when there is a faulty battery/it is discharged. Checking the health/repacing the battery should be the main way to solve this, otherwise an unsafe configuration could cause data loss!):

megacli -LDSetProp -ForcedWB -Immediate -Lall -aAll

Force the BBU to relearn cycle

megacli -AdpBbuCmd -BbuLearn -aAll

Array status

To check the whole array status (ie: we have a RAID10 and we want to check the RAID status we can run)

megacli -LDInfo -L0 -a0

And that would let us know the RAID status and the information about it (and its SPANs)

Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 1.633 TB
Sector Size         : 512
Mirror Data         : 1.633 TB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives per span:2
Span Depth          : 6
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: Yes
Is VD Cached: Yes
Cache Cade Type : Read Only

As we can see in that RAID we have 6 SPANs and it has 2 disks per SPAN, if we are interested in checking the SPAN status (to see which of the SPANs is degraded)

megacli -CfgDsply -a0

There we can see the different SPANs, the number of disks, the status of the SPANs and the information about the disks that belong to each SPAN.

(in this case we have two as it says on PDs: 2 and those disks are Phisicaly Disk 0 and Phisical Disk 1)

SPAN: 5
Span Reference: 0x05
Number of PDs: 2
Number of VDs: 1
Number of dedicated Hotspares: 0
Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 1.633 TB
Sector Size         : 512
Mirror Data         : 1.633 TB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives per span:2
Span Depth          : 6
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: Yes
Is VD Cached: Yes
Cache Cade Type : Read Only
Physical Disk Information:
Physical Disk: 0
Enclosure Device ID: 32
Slot Number: 10
Drive's position: DiskGroup: 0, Span: 5, Arm: 0
Enclosure position: N/A
Device Id: 10
WWN: 5000C50043A97C34
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS

Raw Size: 279.396 GB [0x22ecb25c Sectors]
Non Coerced Size: 278.896 GB [0x22dcb25c Sectors]
Coerced Size: 278.875 GB [0x22dc0000 Sectors]
Sector Size:  0
Firmware state: Online, Spun Up
Device Firmware Level: ES64
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5000c50043a97c35
SAS Address(1): 0x0
Connected Port Number: 0(path0)
Inquiry Data: SEAGATE ST3300657SS     ES646SJ3NDEM
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :38C (100.40 F)
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Port-1 :
Port status: Active
Port's Linkspeed: Unknown
Drive has flagged a S.M.A.R.T alert : No



Physical Disk: 1
Enclosure Device ID: 32
Slot Number: 11
Drive's position: DiskGroup: 0, Span: 5, Arm: 1
Enclosure position: N/A
Device Id: 11
WWN: 5000C50054EA3600
Sequence Number: 4
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 8
Last Predictive Failure Event Seq Number: 37843
PD Type: SAS

Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
Non Coerced Size: 558.411 GB [0x45cd2fb0 Sectors]
Coerced Size: 558.375 GB [0x45cc0000 Sectors]
Sector Size:  0
Firmware state: Online, Spun Up
Device Firmware Level: ES65
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5000c50054ea3601
SAS Address(1): 0x0
Connected Port Number: 0(path0)
Inquiry Data: SEAGATE ST3600057SS     ES656SL4GP4B
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :40C (104.00 F)
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Port-1 :
Port status: Active
Port's Linkspeed: Unknown
Drive has flagged a S.M.A.R.T alert : Yes

Disk status

To check all the disks we can use:

megacli -PDList -aall

Probably, the most wanted fields we are looking at the output from this commands are:

Slot number and device ID

Firmware status (a failed disk will show Failed)

Foreign state: normally it should have "None", unless we insert a disk which has belonged to a different RAID. In that case we have to clear the previous configuration.

Smart status

See SMART#SMART_not_healthy

Controller log output to a file

megacli -AdpEventLog -IncludeDeleted -f log.txt -aALL


Replacing Disks

Automatic array rebuild with a new disk

Normally when a disk fails, and the controller detects it, it gets marked as a bad disks (Firmware status: Failed)

It needs to be replaced, and if it is a brand new disk, as soon as it gets replaced, the controller will start to rebuild the RAID.

By using the command

megacli -PDList -aall

We can identify the disk that failed (look for the one that has Firmware status: Failed) and by looking at its Enclosure devide ID and Decide ID we can form the following command to check the % on the RAID rebuild:

megacli -PDRbld -ShowProg -PhysDrv [32:4] -aALL

Rebuild Progress on Device at Enclosure 32, Slot 4 Completed 2% in 5 Minutes.

Exit Code: 0x00

Once it is finished we will be able to see that the array is fine again by running:

megacli -AdpAllInfo -aALL
        Device Present
                ================
Virtual Drives    : 1
  Degraded        : 0
  Offline         : 0
Physical Devices  : 14
  Disks           : 12
  Critical Disks  : 0
  Failed Disks    : 0

Manually array rebuild with an used disk

If we use a non new disk, which has been part of an array in the past, it will contain metadata and the controller will not start to rebuild the array until we have cleaned that configuration, as a security precaution.

First of all we need to make sure that indeed, the RAID is not rebuilding because there is a foreign configuration on a disk.

We can scan for old configurations:

megacli -CfgForeign -Scan -aALL

There are 1 foreign configuration(s) on controller 0.

In that case we can verify that if it is indeed our new disk:

 megacli -PDList -aall

When looking for our new disk we will realise it will have these fields:

Enclosure Device ID: 32
Device Id: 4
Firmware state: Unconfigured(good), Spun Up
Foreign State: Foreign

So we know that the disk 32:4 is in a good state but it was part of an array before and needs to be cleaned up.

megacli -CfgForeign -Clear -aALL

Foreign configuration 0 is cleared on controller 0.

Exit Code: 0x00

Now we can check if we are good to go:

megacli -CfgForeign -Scan -aALL

There is no foreign configuration on controller 0.

Exit Code: 0x00

We are ready to replace the disk.

First of all we need to identify in which SPAN the old disk was and in which position.

We firstly need to go thru all the SPANs:

megacli -CfgDsply -a0 | grep "Number of Spans"
Number of Spans: 6

And we need to know how many disks per SPAN we should have:

megacli -CfgDsply -a0 | grep "Number of PDs" -m1
Number of PDs: 2

There must be 2 disks per SPAN, so we need to scan all the SPANs to see which one has a disk with no information.

Normally a SPAN will show the Physical Disk information empty if a disk is missing.

In this case we stop at SPAN 2 because the information for Physical Disk 0 (in bold) is missing (compare it with the information it shows for Physical Disk 1).

And the SPAN also shows: State: Degraded

SPAN: 2
Span Reference: 0x02
Number of PDs: 2
Number of VDs: 1
Number of dedicated Hotspares: 0
Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 1.633 TB
Sector Size         : 512
Mirror Data         : 1.633 TB
State               : Degraded
Strip Size          : 256 KB
Number Of Drives per span:2
Span Depth          : 6
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: Yes
Is VD Cached: Yes
Cache Cade Type : Read Only
Physical Disk Information:
Physical Disk: 0



Physical Disk: 1
Enclosure Device ID: 32
Slot Number: 5
Drive's position: DiskGroup: 0, Span: 2, Arm: 1
Enclosure position: N/A
Device Id: 5
WWN: 5000C50043A74B14
Sequence Number: 2
Media Error Count: 12
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS

Raw Size: 279.396 GB [0x22ecb25c Sectors]
Non Coerced Size: 278.896 GB [0x22dcb25c Sectors]
Coerced Size: 278.875 GB [0x22dc0000 Sectors]
Sector Size:  0
Firmware state: Online, Spun Up
Device Firmware Level: ES64
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5000c50043a74b15
SAS Address(1): 0x0
Connected Port Number: 0(path0)
Inquiry Data: SEAGATE ST3300657SS     ES646SJ3MN6X
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :38C (100.40 F)
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Port-1 :
Port status: Active
Port's Linkspeed: Unknown
Drive has flagged a S.M.A.R.T alert : No

So our new disk 32:4 needs to go to SPAN2 and and become Physical Disk 0 so for megacli that means row0 of the SPAN.

megacli -PdReplaceMissing -PhysDrv[32:4] -array2 -row0 -a0

Adapter: 0: Missing PD at Array 2, Row 0 is replaced.

Now we are ready to rebuild the ARRAY.

megacli -PDRbld -Start -PhysDrv[32:4] -a0

Started rebuild progress on device(Encl-32 Slot-4)

And we can check the progress

 megacli -PDRbld -ShowProg -PhysDrv [32:4] -aALL

Rebuild Progress on Device at Enclosure 32, Slot 4 Completed 9% in 2 Minutes.

Note that the ARRAY and the SPAN will not be set to Optimal until the disk has finished. Once it has finished.

Check and clear Preserved Cache

In cases where a disk fails and it is replaced it might happen that the controller still has cached data for said disk. Note that when in this state the controller requires manual intervention to boot and it will prompt for "preserved data cache present".

 FW error description: 
 The current operation is not allowed because the controller has data in cache for offline or missing virtual drives.  

If that's the case it is possible to list all disks with preserved cache:

 # megacli -GetPreservedCacheList -a0
 Adapter #0
 Virtual Drive(Target ID 09): Missing.
 Exit Code: 0x00

And clear the preserved cache for the drive in question:

 # megacli -DiscardPreservedCache  -L09 -a0                
 Adapter #0
 Virtual Drive(Target ID 09): Preserved Cache Data Cleared.
 Exit Code: 0x00


Manually replace multiple disks

The steps to replace multiple failed disks (of different SPANs) are the same as for MegaCli#Manually array rebuild with an used disk however, we need to make sure we do one disk at the time.

Please read the MegaCli#Manually array rebuild with an used disk before continuing here, so you understand what is going to be done now:

Once we have identified the SPANs and disks that need replace, execute the Replace+Rebuild commands one by one, and do not start all of them at the same time.

Wait for each SPAN to be finished before starting with the new one.

Note: the RAID and the SPANs will not be set to Optimal until all the degraded SPANs have been fixed.

There is an example of how to operate multiple failed disks here:

https://phabricator.wikimedia.org/T149099#2751204 and this is how to start replacing each disk: https://phabricator.wikimedia.org/T149099#2762624

Replace individual disks in JBOD

Some hosts (e.g. ms-be) have a raid controller and use all disks in each own raid 0 virtual drive (essentially JBOD). It might happen that the disk shows errors to the OS and the controller thinks the disk itself is fine (i.e. doesn't mark the disk as failed). In these cases we need to fail the disk manually as follows:

 # Identify the Physical Drive ID
 /usr/local/lib/nagios/plugins/get-raid-status-megacli
 === RaidStatus (does not include components in optimal state)
 name: Adapter #0
   Virtual Drive: 16 (Target Id: 16)
   RAID Level: Primary-0, Secondary-0, RAID Level Qualifier-0
   State: Optimal
   Number Of Drives: 1
   Number of Spans: 1
   Current Cache Policy: WriteBack, ReadAheadNone, Cached, No Write Cache if Bad BBU
      Span: 0 - Number of PDs: 1
        PD: 0 Information
        Enclosure Device ID: 32
        Slot Number: 14
        Drive's position: DiskGroup: 16, Span: 0, Arm: 0
        Media Error Count: 2360
        Other Error Count: 1
        Predictive Failure Count: =====> 1 <=====
        Last Predictive Failure Event Seq Number: 4812
        Raw Size: 7.277 TB [0x3a3812ab0 Sectors]
        Firmware state: Online, Spun Up
        Media Type: Hard Disk Device
        Drive Temperature: 41C (105.80 F)

The disk is PD slot 14 and enclosure 32 in adapter 0, therefore:

 # Mark as offline
 megacli -PDOffline -PhysDrv '[32:14]' -a0
 # Locate the drive (blink the led)
 megacli -PDLocate -PhysDrv '[32:14]' -a0

These actions will trigger a task in Phabricator. After replacement the disk can be put back (and clearing the preserved cache first if needed)

 megacli -GetPreservedCacheList -a0
 megacli -DiscardPreservedCache -L'disk_number' -a0
 megacli -CfgEachDskRaid0 WB RA Direct CachedBadBBU -a0

See also