Portal:Cloud VPS/Admin/Runbooks/Kernel panic
Error / Incident
The server had a number of panic / error messages showing up in the dmesg kernel print buffer.
Depending on the error, you may want to take the server out of service immediately (depool).
Debugging
This is quite a generic alert, so there's many options. In general, a kernel panic may indicate any or all of:
- a linux kernel bug
- a hardware problem
- a firmware problem
- other severe misconfiguration
Try to ssh to the host and check the status, for example:
journactl -k --since today
Error-level logs:
journalctl -k -p err
Check all other logs:
journalctl --since today
You may also want to check idrac logs. TODO: add link.
Common issues
Add notes here when you find this issue.
Related information
See https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation
Support contacts
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect or the bridged Telegram group
- Discuss via email after you have subscribed to the cloud@ mailing list
- Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
- Read the News wiki page
Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)
Old incidents
Add your incident here: