Server Out-of-Band Management-Access

Every Fundraising server has an out-of-band management interface. The interface is connected to a network switch, which is connected to the PFW firewall/router and configured as an isolated management subnet. The management network is accessible via tunneled SSH connection through the Fundraising bastion servers.

Dell - DRAC

Connect by SSH

Make sure your ~/.ssh/config has Host config to route the ssh connection through the bastion server, and to set the correct user.

Host *.mgmt.frack.*.wmnet
    User root
    ProxyCommand /usr/bin/ssh -q -W %h:%p frbast.wikimedia.org

Then:

ssh {host}.mgmt.frack.{dc}.wmnet
Password: <first password is bastion SSH 2FA>
Password: <second password is management password>

Useful Commands/Operations

 racadm racdump                 Display Hardware Information and Configuration
 racadm getsysinfo              Display System Information (including host IPs)
 racadm serveraction <action>   Control Server Power (powerup|powerdown|powercycle)

Change DRAC password (one or the other, depending on DRAC version)

racadm config -g cfgUserAdmin -o cfgUserAdminPassword -i 2 <thepassword>
racadm config -g cfgUserAdmin -o cfgUserAdminPassword -i 1 <thepassword>
racadm set iDRAC.Users.2.Password <thepassword>

Set to PXE boot on the next restart (newer hosts):

racadm set iDRAC.ServerBoot.FirstBootDevice PXE

Set to PXE boot on the next restart (older hosts):

racadm config -g cfgServerInfo -o cfgServerBootOnce 1
racadm config -g cfgServerInfo -o cfgServerFirstBootDevice PXE

Configure Network Interface (usually handled by dcops):

racadm setniccfg -s 10.64.40.199 255.255.255.192 10.64.40.193

Command and Hotkeys to Enter/Exit console or BIOS:

enter console: console com2
exit console:  ^\
enter bios:    <esc>2
start pxeboot: ^<esc><2>

UEFI boot settings

 racadm set BIOS.BiosBootSettings.BootMode Uefi
 racadm jobqueue create BIOS.Setup.1-1 -r pwrcycle -s TIME_NOW
 (wait for the box reboot and update its settings, then...)
 racadm set BIOS.NetworkSettings.HttpDev1EnDis Disabled
 racadm set BIOS.NetworkSettings.HttpDev2EnDis Disabled
 racadm set BIOS.NetworkSettings.HttpDev3EnDis Disabled
 racadm set BIOS.NetworkSettings.HttpDev4EnDis Disabled
 racadm set BIOS.NetworkSettings.PxeDev1EnDis Enabled
 racadm set BIOS.NetworkSettings.PxeDev2EnDis Disabled
 racadm set BIOS.NetworkSettings.PxeDev3EnDis Disabled
 racadm set BIOS.NetworkSettings.PxeDev4EnDis Disabled
 (this may not work, you may have to go into BIOS and set it there)
 racadm set BIOS.NetworkSettings.UefiPxeInd NIC.Integrated.1-1-1 Enabled
 racadm set BIOS.BiosBootSettings.UefiBootSeq HardDisk.List.1-1,NIC.Integrated.1-1-1
 racadm jobqueue create BIOS.Setup.1-1 -r pwrcycle -s TIME_NOW

Serial Console settings

 racadm set BIOS.SerialCommSettings.SerialComm OnConRedir
 racadm set BIOS.SerialCommSettings.SerialPortAddress Com2
 racadm set BIOS.SerialCommSettings.RedirAfterBoot Enabled
 racadm jobqueue create BIOS.Setup.1-1 -r pwrcycle -s TIME_NOW

Other/General if not already done

* disable pxeboot for "embedded" NICs
* enable pxeboot for first "integrated" NIC

Supermicro

Connect by IPMI

ipmitool is installed on the bastion servers
mpi is a wrapper script to simplify commands, see: mpi -h
if using ipmitool directly, remember -I and -E (see man page) and don't put the password in the command
optionally set the password as an environment variable, this works for mpi and ipmitool:

read -s -p "IPMI Password: " IPMI_PW
export IPMI_PASSWORD=$IPMI_PW

Serial Over LAN (sol)

for console access to BIOS and redirected COM ports
sol hijacks keyboard input, normal shell escapes won't work
to exit sol, <enter> then wait a second and ~~.
if this ^^^ doesn't work, repeat and/or close your terminal locally
if arrow keys etc stop working, exit/kill the session and 'reset' in terminal
to close a lingering sol session run: mpi payments1010 sol deactivate

BIOS Access

mpi <short hostname> chassis bootdev bios
mpi <short hostname> chassis power cycle
mpi <short hostname> sol activate

BIOS Settings

Supermicro X13SEW-F

  Advanced -> Serial Port Console Redirection -> COM1 / Console Redirection: Enabled
  Boot -> Boot Mode Select: UEFI
  (following options may change after you set boot mode to UEFI and reboot)
  Boot -> FIXED BOOT ORDER Priorities -> Boot Option #1: UEFI Hard Disk
  Boot -> FIXED BOOT ORDER Priorities -> Boot Option #2: UEFI Network