Jump to content

Fundraising/techops/procedures/servers-out of band management access

From Wikitech

Server Out-of-Band Management-Access

Every Fundraising server has an out-of-band management interface. The interface is connected to a network switch, which is connected to the PFW firewall/router and configured as an isolated management subnet. The management network is accessible via tunneled SSH connection through the Fundraising bastion servers.

Connecting by SSH

Make sure your ~/.ssh/config has Host config to route the ssh connection through the bastion server, and to set the correct user.

Host *.mgmt.frack.*.wmnet
    User root
    ProxyCommand /usr/bin/ssh -q -W %h:%p frbast.wikimedia.org

Then:

ssh {host}.mgmt.frack.{dc}.wmnet
Password: <first password is bastion SSH 2FA>
Password: <second password is management password>

Dell DRAC Useful Commands/Operations

  • Display Hardware Information and Configuration
racadm racdump
  • Display System Information (including host IPs)
racadm getsysinfo
  • Control Server Power
racadm serveraction <action>
possible actions include powerup, powerdown, and powercycle
  • Change DRAC password (one or the other, depending on DRAC version)
racadm config -g cfgUserAdmin -o cfgUserAdminPassword -i 2 <thepassword>
racadm config -g cfgUserAdmin -o cfgUserAdminPassword -i 1 <thepassword>
racadm set iDRAC.Users.2.Password <thepassword>
  • Set to PXE boot on the next restart (newer hosts):
racadm set iDRAC.ServerBoot.FirstBootDevice PXE
  • Set to PXE boot on the next restart (older hosts):
racadm config -g cfgServerInfo -o cfgServerBootOnce 1
racadm config -g cfgServerInfo -o cfgServerFirstBootDevice PXE
  • Configure Network Interface (usually handled by dcops):
racadm setniccfg -s 10.64.40.199 255.255.255.192 10.64.40.193
  • Command and Hotkeys to Enter/Exit console or BIOS:
enter console: console com2
exit console:  ^\
enter bios:    <esc>2
start pxeboot: ^<esc><2>
  • Fix Post-Boot Console Redirection:
System BIOS Settings > Serial Communication > Redirection After Boot <Enabled>