Error / Incident
This alert fires when the Openstack API performance is detected to be "poor". By the time of this writing, this value has been established as the 12h average response time to be > 1 second.
The performance problem is likely affecting the Openstack control plane and making it unrealiable, slow, and/or prone to transient failures.
This alert is based on the prometheus metrics of HAproxy. You can check this grafana dashboard:
As of this writing, there should be one primary HAproxy node, that you can check with:
user@laptop:~ $ host openstack.eqiad1.wikimediacloud.org openstack.eqiad1.wikimediacloud.org is an alias for cloudcontrol1007.wikimedia.org. [..]
You can try by hand some openstack commands and see how they behave:
user@cloudrabbit1007:~$ sudo wmcs-openstack endpoint list [..] user@cloudrabbit1007:~$ sudo wmcs-openstack server list --all-projects [..] user@cloudrabbit1007:~$ sudo wmcs-openstack zone list --all-projects [..] user@cloudrabbit1007:~$ sudo wmcs-openstack subnet list --all-projects [..]
You can check basic server stats, like CPU, memory, load average, etc. Try using tools like
htop and similar.
Some common operations to handle this alert.
In the past, restarting all the openstack control plane has solved the performance problem for the most part.
There is an automated cookbook to do it:
user@laptop:~ $ cookbook wmcs.openstack.restart_openstack --task-id T11111 --cluster-name eqiad1 --all [..]
Increase number of worker
In the past, increasing the number of API workers has helped accommodate additional load.
Example ticket: T336379 - Openstack API slowdowns.
- phab:T345084 (open task to track this recurring issue)
- In the past, we know prometheus scrapping of the openstack metrics have causes performance to suffer.
See for example phabricator ticket: T335943