This page contains information about our HAProxy configuration in the CloudVPS environment.

OpenStack API Endpoints

Backend HA Status

Backend services that are full active / active will automatically be pooled and depooled by HAProxy. As long as there is one instance available there will be no interruption in service.

EQIAD1

Service	active backends	standby backends	notes
Designate	cloudservices1003 cloudservices1004	NA
Glance API	cloudcontrol1003	cloudcontrol1004	Waiting on shared storage (CEPH) for full active/active
Glance Registry	cloudcontrol1003	cloudcontrol1004	Waiting on shared storage (CEPH) for full active/active
Keystone	cloudcontrol1003 cloudcontrol1004	NA
Neutron	cloudcontrol1003 cloudcontrol1004	NA
Nova	cloudcontrol1003 cloudcontrol1004	NA

Port Mappings

Service	Backend Port	Frontend Port
Designate API	9001	9001
Glance API	19292	9292
Glance Registry	19191	9191
Keystone Admin	15000	5000
Keystone Public	45357	35357
Neutron	19696	9696
Nova API	18774	8774
Nova Metadata	18775	8775

Load Balancing and Session Handling

Backend connections for each service are dynamically directed to the server with the least amount of connections using the `leastconn` load balancing algorithm.

Session persistence is maintained with HAProxy stick tables using the IP address of the client. This will ensure new client connections get forwarded to the same server every time.

Monitoring

Icinga

Icinga is configured with the following HAProxy process checks:

nrpe_check!check_haproxy_alive!10
nrpe_check!check_haproxy!10

Prometheus

The prometheus haproxy exporter is collecting haproxy backend and frontend metrics directly from haproxy through the URL http://localhost:9900/stats;csv. This URL is read-only and only provides statistic information through the loopback interface.

Grafana dashboard: https://grafana.wikimedia.org/d/tanisM2Zz/wmcs-openstack-eqiad1-api-stats

Troubleshooting

Monitoring

HAProxy statistics and metrics can be queried through the "stats socket" at `/run/haproxy/haproxy.sock`.

Process information:

echo "show info" | sudo socat /run/haproxy/haproxy.sock stdio

Statistics:

echo "show stat" | sudo socat /run/haproxy/haproxy.sock stdio

Log files

/var/log/haproxy/haproxy.log

Primary Failover

Design

Each controller is running HAProxy with no cross dependencies on other controllers or HAProxy instances. Each load balancer is operating on its own with no shared information like client sessions between them.

DNS

Each region has a dedicated FQDN (openstack.<region>.wikimediacloud.org) which maps to one of the OpenStack controllers running HAProxy. In the event of failure or maintenance this DNS entry will need to be remapped to the other controller in the operations/dns git repository at https://gerrit.wikimedia.org/r/admin/projects/operations/dns