Platform-specific documentation/HP DL3N0 Gen9

From Wikitech
Jump to navigation Jump to search

We have a few of these which are used as Labs virtualization servers: labvirt1010 and 1011. These are also used in other systems. Due to being 'Gen 9' systems, they are different from the Gen8 (like the older labvirt nodes) in several ways. Mercifully, the iLO (aka 'mgmt') console still works the same as the DL3N0 boxes, so refer to that page for mgmt commands.

system setup

Enter the system setup tool by pressing esc-9 during boot. The terminal emulation of this tool is lousy, so things will scrawl all over your screen and generally be hard to use.

Setting proper power option

In bios:

* select service options

* Set Processor Power Monitoring and choose disabled

* Press enter, ignore warning message regarding modification by pressing enter again.  Select disabled and press enter again.

Embedded user partition

Make sure that this setting is turned off, otherwise partman will hate you:

 System Configuration -> BIOS/Platform Configuration (RBSU) -> System Options -> USB Options -> Embedded User Partition = Disabled

10g nics disabled

Currently we aren't using the 10g network adapter. Disable it in the bios so that the system doesn't try to use it for net boot.

Enable virtualization

Virtualization is turned off by default. Make this change to enable virtualization:

 System Configuration -> BIOS/Platform Configuration (RBSU) -> System Options -> Virtualization Options -> Virtualization Technology = Enabled


System Configuration -> BIOS/Platform Configuration (RBSU) -> System Options -> Virtualization Options -> Intel(R) VT-d  = Enabled


System Configuration -> BIOS/Platform Configuration (RBSU) -> System Options -> Virtualization Options -> SR-IOV = Enabled

Allow netboot

Systems arrive configured with UEFI boot mode, which doesn't work with our existing netboot system. To enable pxe boot, change

 System Configuration -> BIOS/Platform Configuration (RBSU) -> Boot Options -> Boot Mode = 'Legacy BIOS Mode'

Drive config

Each server as 8 1.2 Tb SSD drives. The need to be configured as one big hardware Raid 10 -- modifying the raid config is possible through management console, the text version takes a while to load and will display no such device on the screen while loading. Once loaded you are dropped into hpssacli like you would on linux (it is in fact loading linux)

ms-be RAID0 config

An easy way to configure swift backend ms-be machines disks all in raid0 using the console above (order is important )

First, reboot the system and during reboot Press 'ESC+9' to enter for System Utilities. Once in the System Utilities, select System Configuration then Slot 3 : Smart Array P840 Controller. Select Exit and launch HP Smart Storage Administrator(HPSSA). At the next step, an error message will appear 'error: no such device: EMBEDDED250.' there is nothing to do at this point, but wait for the hpssacli prompt (==>)

 set target controller slot=3
 array all delete forced
 create type=arrayr0 drivetype=ss_sata
 create type=arrayr0 drivetype=sata

Mark a disk as failed

It might happen that Linux detects errors while writing to a disk but the raid controller itself doesn't see the disk as failed (e.g. https://phabricator.wikimedia.org/T163690). In these cases it is useful to forcefully mark the physical drive as failed as follows:

 set target controller slot=3
 pd all show
 # take note of the disk e.g. 1I:1:5
 pd DISK modify disablepd forced

To reenable the LD (not the PD) after the disk has been swapped:

 ld NUMBER modify reenable

Blink disk led

Via hpssacli:

 set target controller slot=3
 pd DISK modify led=on

ACPI Errors

On first install and after the first puppet run there might be messages similar to this showing up on console:

 ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20160831/exfield-427)
 ACPI Error: Method parse/execution failed [\_SB.PMI0._PMM] (Node ffff8a523f04f2f8), AE_AML_BUFFER_LIMIT (20160831/psparse-543)
 ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20160831/power_meter-338)

This is related to the "power meter" ACPI module loaded, we blacklist the module since https://gerrit.wikimedia.org/r/#/c/356422/ and thus a reboot will make the message disappear.

complete system specs

 (Virtualization Server) 2xE5-2697v3 (2.6GHz/14-C), 384GB RAM, 2x10G, P440
 HP ProLiant DL360 Gen9 8SFF Configure-to-order Server
 HP DL360 Gen9 Intel Xeon E5-2697v3 (2.6GHz/14-core/35MB/145W) FIO Processor Kit
 HP DL360 Gen9 Intel Xeon E5-2697v3 (2.6GHz/14-core/35MB/145W) Processor Kit
 12 x HP 32 GB (1x32 GB) Dual Rank x4 DDR4-2133 CAS-15-15-15 Registered Memory Kit
 HP FLEXFABRIC 10GB 2P 534FLR-SFP+ ADPTR
 HP Smart Array P440ar/2GB FBWC 12Gb 2-ports Int FIO SAS Controller
 HP 800W FS Plat Ht Plg Power Supply Kit
 8 x Intl DC S3510 1.20 TB 2.5in internal Solid State Drive -SATA - 500 MB/s Maximimum Read Transfer Rate - 440 MB/s Maximum Write Transfer Rate -1 Pack - 256-bit Encryption Standard

RAID controller firmware upgrade

See Platform-specific_documentation/HP_DL3N0#RAID_controller_firmware_upgrade

Platform-specific documentation/HP ProLiant DL380 Gen10

RAID setup

The ms-be systems RAID configuration is each disk in its own RAID 0 Starting from the SSD disks first. So the ms-be systems in general comes with a total of 14 disks. Counting from 0 to 13, the ssd's are in slot 12 and 13. You need to create first a RAID 0 for the first SSD disk in slot 12 then another RAID 0 for the SSD in slot 13 so that each SSD's will take as name sda and sdb. After that, do the same for the other 12 disks. At the end you will have:

Array A Array B Array C Array D Array E Array F Array G Array H Array I Array J Array K Array L Array M Array N

Array A being the SSD in slot 12 and Array B the SSD in slot 13

once in BIOS go to "system Configuration" - "Embedded RAID 1 : HPE Smart Array P816i-a SR Gen10 " - "Array Configuration " - "Create Array "

BIOS configuration =

  • Disable Internal SD card slot

If this is not disable the installation will failed at disk partition.

Once in BIOS go to "System Configuration" - "System utilities" - "System Options" - "USB Options" - " Internal SD card slot" and change it from enabled to disabled

  • Set boot option to "Legacy BIOS mode"

Once in BIOS go to "System Configuration" - "System utilities" - "System Options" -" Boot options" change mode mode from UEFI mode to Legacy BIOS mode