Anycast recursive DNS
In order to improve resiliency of recursive DNS, this setup leverages BGP and anycast.
Task: https://phabricator.wikimedia.org/T186550
CR: https://gerrit.wikimedia.org/r/c/operations/puppet/+/397723/
Limitation of a non-anycast setup
- Some services don't fail over fast enough to the 2nd server listed on resolv.conf when one fails
- If the two servers of a site (or the whole site) fails, servers relying on them will experience an outage
- LVS/pybal depends on DNS and thus making it a chicken/egg problem
Configuration
Server side
modules/role/manifests/dnsrecursor.pp include ::profile::bird::anycast
hieradata/role/common/recursor.yaml (global)
profile::bird::advertise_vips:
recdns.anycast.wmnet:
address: 10.3.0.1 # VIP to advertise (limited to a /32)
check_cmd: '/usr/lib/nagios/plugins/check_dns_query -H 10.3.0.1 -l -d www.wikipedia.org -t 1'
service_type: recdns
check_cmd
In this case we re-use an Icinga NRPE check, installed on all the servers:
/usr/lib/nagios/plugins/check_dns_query -H 10.3.0.1 -l -d www.wikipedia.org -t 1
Troubleshooting
Know which server a client is redirected to
$ dig @10.3.0.1 CHAOS TXT id.server. +short
Ensure all servers can reach the VIP
bblack@cumin1001:~$ sudo cumin '*' 'dig @10.3.0.1 CHAOS TXT id.server. +short'
Failure tests
Single local recursor failure
bblack@backup2001:~$ while [ 1 ]; do echo ======; date; dig @10.3.0.1 CHAOS TXT id.server. +short; sleep 1; done
"so I can see a result once a second, I've tried stopping just healthchecker, stopping or killing the recursor, etc"
Traffic routes to the one working local node within the second.
Double local recursor failure
Eg. take down dns2001/dns2002
Request end up on dns1001/dns1002
Limitations
- If the DNS recursors have the anycast VIP as only resolver in resolv.conf, then processes depending on DNS will fail until pdnsd starts as they will try to connect to the local recdns service instead of being routed to the closest server.
- Workaround is to either hardcode real recursors IPs or have a daemon that remove the VIP loopback
Future evolution
- Add Icinga monitoring to check local recursors work (eg. Icinga check on bastX hosts that check it's dnsX that reply and not dnsY)