Portal:Cloud VPS/Admin/DNS

From Wikitech
Jump to navigation Jump to search
See also: Portal:Cloud VPS/Infrastructure#dns

Private DNS

Within Cloud VPS, each instance has a name like <instancename>.<projectname>.eqiad.wmflabs. For historical reasons we also create <instancename>.eqiad.wmflabs DNS entries for each instance. This legacy behavior may be discontinued in the future. CloudVPS is not just one big flat zone in the same way production is, it is broken into tenants with different access restrictions.

There is a special private domain which is svc.eqiad.wmflabs. which is intended to hold service FQDNs not associated with virtual machines or a specific Cloud VPS project.

Public DNS

Public DNS (e.g. tools-login.wmflabs.org) is currently handled by labs-ns0 and labs-ns1 running pdns with designate.

Designate

Cloud VPS DNS is PowerDNS, backed by a database controlled by Designate.

When a new instance is created, a DNS A record like this is created in Designate (under the special noauth-project tenant which novaobserver cannot access?):

; <<>> DiG 9.11.3-1ubuntu1.2-Ubuntu <<>> novaproxy-01.project-proxy.eqiad.wmflabs @labs-ns0.wikimedia.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21651
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 2800
;; QUESTION SECTION:
;novaproxy-01.project-proxy.eqiad.wmflabs. IN A
;; ANSWER SECTION:
novaproxy-01.project-proxy.eqiad.wmflabs. 60 IN	A 10.68.21.68
;; Query time: 91 msec
;; SERVER: 208.80.155.117#53(208.80.155.117)
;; WHEN: Wed Nov 07 12:48:08 GMT 2018
;; MSG SIZE  rcvd: 85

The public DNS stuff lives under the wmflabsdotorg tenant:

krenair@shinken-02:~$ OS_PROJECT_ID=wmflabsdotorg openstack zone list
+--------------------------------------+---------------------------------+---------+------------+--------+--------+
| id                                   | name                            | type    |     serial | status | action |
+--------------------------------------+---------------------------------+---------+------------+--------+--------+
| 553ef162-add7-4a5c-b115-9cabca662746 | wmflabs.org.                    | PRIMARY | 1541584686 | ACTIVE | NONE   |
| ef825cf2-db2d-4480-ad33-2c19b0a188dc | wmflabsdotorg.wmflabs.org.      | PRIMARY | 1535835306 | ACTIVE | NONE   |
| 933a78c2-3d8d-4ee1-bbef-9ab30be5f972 | 128-25.155.80.208.in-addr.arpa. | PRIMARY | 1535835306 | ACTIVE | NONE   |
| e07defe3-6d08-4f37-bd7d-1bdc1c45d7e8 | 56.15.185.in-addr.arpa.         | PRIMARY | 1541514722 | ACTIVE | NONE   |
+--------------------------------------+---------------------------------+---------+------------+--------+--------+

When floating IPs are allocated and assigned, and DNS records pointed at them like so:

krenair@shinken-02:~$ OS_PROJECT_ID=wmflabsdotorg openstack recordset show wmflabs.org. ntp-01.wmflabs.org.
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| action      | NONE                                 |
| created_at  | 2018-11-02T20:34:18.000000           |
| description | time server for cloud instances      |
| id          | 5812d178-b7a8-4168-bce7-5e064b411f82 |
| name        | ntp-01.wmflabs.org.                  |
| records     | 185.15.56.3                          |
| status      | ACTIVE                               |
| ttl         | 3600                                 |
| type        | A                                    |
| updated_at  | None                                 |
| version     | 1                                    |
| zone_id     | 553ef162-add7-4a5c-b115-9cabca662746 |
+-------------+--------------------------------------+

Alex's dns-floating-ip-updater.py script creates something like this:

krenair@shinken-02:~$ OS_PROJECT_ID=wmflabsdotorg openstack recordset show 56.15.185.in-addr.arpa. 3.56.15.185.in-addr.arpa.
+-------------+---------------------------------------------------------------------------+
| Field       | Value                                                                     |
+-------------+---------------------------------------------------------------------------+
| action      | NONE                                                                      |
| created_at  | 2018-11-02T20:41:47.000000                                                |
| description | MANAGED BY dns-floating-ip-updater.py IN PUPPET - DO NOT UPDATE OR DELETE |
| id          | bad4939b-aaf3-4bef-8878-2351b8944b76                                      |
| name        | 3.56.15.185.in-addr.arpa.                                                 |
| records     | ntp-01.wmflabs.org.                                                       |
|             | ntp-01.cloudinfra.wmflabs.org.                                            |
| status      | ACTIVE                                                                    |
| ttl         | None                                                                      |
| type        | PTR                                                                       |
| updated_at  | 2018-11-02T21:01:45.000000                                                |
| version     | 2                                                                         |
| zone_id     | e07defe3-6d08-4f37-bd7d-1bdc1c45d7e8                                      |
+-------------+---------------------------------------------------------------------------+

and usually instance-$instance.$project.wmflabs.org records to make it obvious which IP is served by which host.

Restarting PowerDNS

this section was written for LDAP integration but may not be relevant with Designate (?)

PowerDNS copes very poorly with interruptions in ldap service. Anytime opendj restarts, pdns needs to be restarted as well. So, to refresh either service (ldap or dns):

   $ sudo service opendj restart (on nembus and/or neptunium)
   $ sudo service pdns restart (on virt1000 and labcontrol2001)

Creating records by hand

Designate records can be created by hand using the API or the CLI.

root@cloudcontrol1003:~# designate --all-tenants --os-project-name admin record-create 16.172.in-addr.arpa. \
>  --name 1.0.16.172.in-addr.arpa. --type PTR --data cloudinstances2b-gw.svc.eqiad.wmflabs. \
>  --description "Neutron virtual router. Record created by hand"
+-------------+------------------------------------------------+
| Field       | Value                                          |
+-------------+------------------------------------------------+
| description | Neutron virtual router. Record created by hand |
| type        | PTR                                            |
| created_at  | 2018-11-30T13:26:47.000000                     |
| updated_at  | None                                           |
| domain_id   | 6990e139-49e6-466c-9421-46cf45f05842           |
| priority    | None                                           |
| ttl         | None                                           |
| data        | cloudinstances2b-gw.svc.eqiad.wmflabs.       |
| id          | 855a3d4d-4caf-4652-b897-d6559a64bb4e           |
| name        | 1.0.16.172.in-addr.arpa.                       |
+-------------+------------------------------------------------+
root@cloudcontrol1003:~# designate --all-tenants --os-project-name admin record-create svc.eqiad.wmflabs. \
>  --name cloudinstances2b-gw.svc.eqiad.wmflabs. --type A --data 172.16.0.1 --description "Neutron virtual router. Record created by hand"
+-------------+------------------------------------------------+
| Field       | Value                                          |
+-------------+------------------------------------------------+
| description | Neutron virtual router. Record created by hand |
| type        | A                                              |
| created_at  | 2018-11-30T13:23:02.000000                     |
| updated_at  | None                                           |
| domain_id   | 114f1333-c2c1-44d3-beb4-ebed1a91742b           |
| priority    | None                                           |
| ttl         | None                                           |
| data        | 172.16.0.1                                     |
| id          | a92d47c2-aec3-4777-ac9f-cb246b3c9e13           |
| name        | cloudinstances2b-gw.svc.eqiad.wmflabs.         |
+-------------+------------------------------------------------+

Detecting leaked records

In some cases, leaked DNS records may happen. We have a custom script to detect/correct them: dnsleaks.py. If this script is run with the --delete argument, it will delete leaked records, which is usefull if there are many of them.

root@cloudcontrol1003:~# /root/novastats/dnsleaks.py
A record for huggle-pg.huggle.eqiad.wmflabs. has multiple IPs: ['10.68.17.231', '172.16.2.31']
This needs cleanup but that isn't implemented and almost never happens.
[...]
a6cf149d-d90f-401c-a380-a83b65a69d79 is linked to missing instance cloudinstances2b-gw.admin.eqiad.wmflabs.
[...]
PTR e06fdc2d-2bcd-40ec-b416-ca4d63f0dce2 is linked to missing instance ci-jessie-wikimedia-1099005.contintcloud.eqiad.wmflab