Portal:Cloud VPS/Admin/Runbooks/OpenstackAPIResponse

From Wikitech
The procedures in this runbook require admin permissions to complete.

Error / Incident

This alert fires when the Openstack API performance is detected to be "poor". By the time of this writing, this value has been established as the 12h average response time to be > 1 second.

The performance problem is likely affecting the Openstack control plane and making it unrealiable, slow, and/or prone to transient failures.


This alert is based on the prometheus metrics of HAproxy. You can check this grafana dashboard:


As of this writing, there should be one primary HAproxy node, that you can check with:

user@laptop:~ $ host openstack.eqiad1.wikimediacloud.org
openstack.eqiad1.wikimediacloud.org is an alias for cloudcontrol1007.wikimedia.org.

You can try by hand some openstack commands and see how they behave:

user@cloudrabbit1007:~$ sudo wmcs-openstack endpoint list
user@cloudrabbit1007:~$ sudo wmcs-openstack server list --all-projects
user@cloudrabbit1007:~$ sudo wmcs-openstack zone list --all-projects
user@cloudrabbit1007:~$ sudo wmcs-openstack subnet list --all-projects

You can check basic server stats, like CPU, memory, load average, etc. Try using tools like htop and similar.


Some common operations to handle this alert.

Services restart

In the past, restarting all the openstack control plane has solved the performance problem for the most part.

There is an automated cookbook to do it:

user@laptop:~ $ cookbook wmcs.openstack.restart_openstack --task-id T11111 --cluster-name eqiad1 --all

Increase number of worker

In the past, increasing the number of API workers has helped accommodate additional load.

Example ticket: T336379 - Openstack API slowdowns.

Old incidents

  • phab:T345084 (open task to track this recurring issue)
  • In the past, we know prometheus scrapping of the openstack metrics have causes performance to suffer.

See for example phabricator ticket: T335943