Portal:Cloud VPS/Admin/Haproxy

From Wikitech
Jump to navigation Jump to search

This page contains information about our HAProxy configuration in the CloudVPS environment.

OpenStack API Endpoints

Backend HA Status

Backend services that are full active / active will automatically be pooled and depooled by HAProxy. As long as there is one instance available there will be no interruption in service.

EQIAD1

Service active backends standby backends notes
Designate cloudservices1003
cloudservices1004
NA
Glance API cloudcontrol1003 cloudcontrol1004 Waiting on shared storage (CEPH) for full active/active
Glance Registry cloudcontrol1003 cloudcontrol1004 Waiting on shared storage (CEPH) for full active/active
Keystone cloudcontrol1003
cloudcontrol1004
NA
Neutron cloudcontrol1003
cloudcontrol1004
NA
Nova cloudcontrol1003
cloudcontrol1004
NA

Port Mappings

Service Backend Port Frontend Port
Designate API 9001 9001
Glance API 19292 9292
Glance Registry 19191 9191
Keystone Admin 15000 5000
Keystone Public 45357 35357
Neutron 19696 9696
Nova API 18774 8774
Nova Metadata 18775 8775

Load Balancing and Session Handling

Backend connections for each service are dynamically directed to the server with the least amount of connections using the `leastconn` load balancing algorithm.

Session persistence is maintained with HAProxy stick tables using the IP address of the client. This will ensure new client connections get forwarded to the same server every time.

Monitoring

Icinga

Icinga is configured with the following HAProxy process checks:

  • nrpe_check!check_haproxy_alive!10
  • nrpe_check!check_haproxy!10

Prometheus

The prometheus haproxy exporter is collecting haproxy backend and frontend metrics directly from haproxy through the URL http://localhost:9900/stats;csv. This URL is read-only and only provides statistic information through the loopback interface.

Grafana dashboard: https://grafana.wikimedia.org/d/tanisM2Zz/wmcs-openstack-eqiad1-api-stats

Troubleshooting

Monitoring

HAProxy statistics and metrics can be queried through the "stats socket" at `/run/haproxy/haproxy.sock`.

Process information:

echo "show info" | sudo socat /run/haproxy/haproxy.sock stdio

Statistics:

echo "show stat" | sudo socat /run/haproxy/haproxy.sock stdio

Log files

  • /var/log/haproxy/haproxy.log

Primary Failover

Design

Each controller is running HAProxy with no cross dependencies on other controllers or HAProxy instances. Each load balancer is operating on its own with no shared information like client sessions between them.

DNS

Each region has a dedicated FQDN (openstack.<region>.wikimediacloud.org) which maps to one of the OpenStack controllers running HAProxy. In the event of failure or maintenance this DNS entry will need to be remapped to the other controller in the operations/dns git repository at https://gerrit.wikimedia.org/r/admin/projects/operations/dns

Openstack HAproxy.png