Portal:Cloud VPS/Admin/Neutron LVS

From Wikitech

This page contains information on running LVS in our CloudVPS environment using Neutron.

Due to the nature and complexity of this use case, this is an experimental setup, and not a public end-user offered service.

Experimental setup

To test LVS-DR, ldirectord was used to manage ipvsadm configuration. I didn't check any actual ops/puppet.git LVS roles to be able to simplify the setup. I didn't test with floating IPs as VIPs (public addresses) to simplify the setup.

  • Create a Cloud VPS project lvs-test in the codfw1dev-r region
  • Create 3 VM instances in the lvs-test project: lvs-server (172.16.128.214), lvs-backend-01 (172.16.128.212) and lvs-backend-02 (172.16.128.213).
  • Create and assign an additional IP address for lvs-server: 172.16.128.211 (this will be our VIP). nova add-fixed-ip <server> <network-id>
  • SSH to VMs and install basic packages: tcpdump, netcat, ipvsadm, ldirectord.
  • Add VIP to the lo interfaces on backend servers ip addr add 172.16.128.211/32 dev lo and to eth0 in the LVS server.
  • Configure ldirectord with a very basic config (in the lvs-server VM) /etc/ha.d/ldirectord.cf:
checktimeout=3
checkinterval=1
autoreload=yes
quiescent=no

virtual=172.16.128.211:80
        servicename=test
        comment=test
        real=172.16.128.212:80 gate
        real=172.16.128.213:80 gate
        service=http
        scheduler=rr
        protocol=tcp
        checktype=ping
  • Check resulting ipvsadm configuration:
root@lvs-server:~# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.128.211:80 rr
  -> 172.16.128.212:80            Route   1      0          0         
  -> 172.16.128.213:80            Route   1      0          0
  • Start testing, from the Neutron server, nothing works with LVS yet (here, the Neutron server acts as the external client):
aborrero@labtestneutron2001:~$ nc 172.16.128.211 80
^C
  • Now (WARNING) beging disabling security features for the involved ports:
root@labtestcontrol2003:~# neutron port-list | egrep 211\|212\|214\|213
| 64d95341-9f1c-469b-a69a-da732b423c0d |                      | fa:16:3e:5e:9e:f0 | {"subnet_id": "7adfcebe-b3d0-4315-92fe-e8365cc80668", "ip_address": "172.16.128.213"} |
| 7306ee5f-42ca-43aa-a25a-9993c899c428 |                      | fa:16:3e:19:48:05 | {"subnet_id": "7adfcebe-b3d0-4315-92fe-e8365cc80668", "ip_address": "172.16.128.211"} |
|                                      |                      |                   | {"subnet_id": "7adfcebe-b3d0-4315-92fe-e8365cc80668", "ip_address": "172.16.128.214"} |
| b166d7aa-3b8f-49f0-a220-b341bda10f4a |                      | fa:16:3e:1f:35:9a | {"subnet_id": "7adfcebe-b3d0-4315-92fe-e8365cc80668", "ip_address": "172.16.128.212"} |
  • disable security group (iptables rules) and port-security (ebtables rules) for all instances Enable the VIP address to circulate in each VM ports:
user@cloudcontrol1004:~$ sudo wmcs-openstack port set --allowed-address ip-address=172.16.128.211 $PORT_UUID_VM1
[..]
user@cloudcontrol1004:~$ sudo wmcs-openstack port set --allowed-address ip-address=172.16.128.211 $PORT_UUID_VM2
[..]
  • check if LVS works now:
aborrero@labtestneutron2001:~ 1 $ nc 172.16.128.211 80
(UNKNOWN) [172.16.128.211] 80 (http) : Connection refused
aborrero@labtestneutron2001:~ 1 $ nc 172.16.128.211 80
(UNKNOWN) [172.16.128.211] 80 (http) : Connection refused

root@lvs-server:~# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.128.211:80 rr
  -> 172.16.128.212:80            Route   1      1          0         
  -> 172.16.128.213:80            Route   1      1          0

root@lvs-backend-01:~# tcpdump -n -i any tcp port 80
12:28:42.570302 IP 10.192.20.4.41844 > 172.16.128.211.80: Flags [S], seq 2177770907, win 29200, options [mss 1460,sackOK,TS val 2027567397 ecr 0,nop,wscale 9], length 0
12:28:42.570333 IP 172.16.128.211.80 > 10.192.20.4.41844: Flags [R.], seq 0, ack 2177770908, win 0, length 0

root@lvs-backend-02:~# tcpdump -n -i any tcp port 80
12:28:40.271707 IP 10.192.20.4.41842 > 172.16.128.211.80: Flags [S], seq 3771176807, win 29200, options [mss 1460,sackOK,TS val 2027567281 ecr 0,nop,wscale 9], length 0
12:28:40.271740 IP 172.16.128.211.80 > 10.192.20.4.41842: Flags [R.], seq 0, ack 3771176808, win 0, length 0
  • it works!

Conclusions

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Neutron_LVS&oldid=1870591"