For more information about Wikimedia's LVS setup in general, see LVS.
At this moment, just a few features distinguish it from lvsmon:
- It's using asynchronous communication, and thus runs all checks in parallel instead of sequentially
- It has an extra monitoring method called IdleConnection, which keeps an idle connection open to all squids, and therefore notices immediately when the Squid processes are shut down / crashing
- It can fetch server lists over HTTP as well as from the local filesystems
...but I intend to polish it more, and extend it with useful things.
The script is in Wikimedia's Git repo "operations/debs", subdirectory pybal.
PyBal is currently installed on our LVS hosts, in directory /usr/sbin. Start or stop it by
Configuration is in /etc/pybal/. pybal.conf defines the LVS service parameters
The list of pooled hosts resides wherever the
pybal::web class is installed via puppet, under the directory
/srv/pybal-config and it will be reachable via the internal address
http://configuration-master.$site.wmnet/pybal, with one file per LVS service. Attributes:
- weight: a larger number means that more requests get sent to this server in comparison with others
- enabled: either True or False, depending on whether you want requests to be sent to this server
The format should be fairly self explanatory; the files more or less use Python assignment / dictionary syntax.
PyBal supports multiple LVS services through a single instance and configuration file pybal.conf, e.g.:
[text] protocol = tcp ip = 220.127.116.11 port = 80 scheduler = wlc config = file:///etc/pybal/text-squids [images] protocol = tcp ip = 18.104.22.168 port = 80 scheduler = wlc config = file:///etc/pybal/upload-squids
Beware, the code as checked out from git has DryRun = True set in ipvs.py, meaning that it will not modify any actual IPVS state but only show the commands for debugging. This should be changed to a command line option, but for now edit that file to DryRun = False.
The configuration files are generated via puppet.
New PyBal releases can be tested on pybal-test200.codfw.wmnet. The systems are deployed with role(pybaltest). Configuration example:
# /etc/pybal/pybal.conf on pybal-test2001 [global] bgp = yes bgp-local-asn = 64496 bgp-peer-address = 10.192.16.140 #bgp-as-path = 64460 bgp-nexthop-ipv4 = 10.192.16.139 bgp-nexthop-ipv6 = 2620:0:860:101:10:192:1:3 instrumentation = yes instrumentation_ips = [ '127.0.0.1', '::1', '10.192.16.139' ] # Lower is prefered bgp-med = 50 # Service definition [textlb6_80] protocol = tcp ip = 2620:0:860:ed1a::1 port = 80 scheduler = sh config = etcd://conf2001.codfw.wmnet/conftool/v1/pools/codfw/cache_text/varnish-fe/ depool-threshold = .5 monitors = ["IdleConnection"] # IdleConnection monitor configuration idleconnection.max-delay = 300 idleconnection.timeout-clean-reconnect = 3
# /etc/pybal/pybal.conf on pybal-test2003 [global] bgp = yes bgp-local-asn = 64496 bgp-peer-address = 10.192.16.140 #bgp-as-path = 64460 bgp-nexthop-ipv4 = 10.192.16.141 bgp-nexthop-ipv6 = 2620:0:860:101:10:192:1:3 instrumentation = yes instrumentation_ips = [ '127.0.0.1', '::1', '10.192.16.141' ] #Lower is prefered bgp-med = 100 # Service definition [...]
A Quagga instance is installed on pybal-test2002 and can be used to test the BGP component of PyBal:
log file /var/log/quagga/quagga.log ! debug zebra rib debug bgp events debug bgp updates debug bgp zebra ! password SECRET ! interface eth0 ipv6 nd suppress-ra ! interface lo ! router bgp 64460 bgp router-id 10.192.16.140 no bgp default ipv4-unicast network 127.0.0.2/32 neighbor 10.192.16.139 remote-as 64496 neighbor 10.192.16.139 description PyBal on pybal-test2001 neighbor 10.192.16.139 activate neighbor 10.192.16.139 prefix-list NONE out neighbor 10.192.16.141 remote-as 64496 neighbor 10.192.16.141 description PyBal on pybal-test2003 neighbor 10.192.16.141 activate neighbor 10.192.16.141 prefix-list NONE out ! address-family ipv6 network 2620:0:860:102::/64 neighbor 10.192.16.139 activate neighbor 10.192.16.141 activate exit-address-family ! ip prefix-list NONE seq 5 deny any ! ip forwarding ipv6 forwarding ! line vty !
The IPv4 routing table can be inspected with:
vtysh -c 'show ip route'
Similarly, to inspect the IPv6 routing table:
vtysh -c 'show ipv6 route'
- lvsmon: The predecessor to PyBal