Portal:Cloud VPS/Admin/Designate

From Wikitech
Jump to navigation Jump to search

This page contains information on how to operate Desginate, which is a key component in our DNS setup.

DNS servers

See DNS setup, servers section.

Operations

Some notes on operating the DNS service provided by Designate.

Context

Cloud VPS DNS is PowerDNS, backed by a database controlled by Designate.

When a new instance is created, a DNS A record like this is created in Designate (under the special noauth-project tenant which novaobserver cannot access?):

; <<>> DiG 9.11.3-1ubuntu1.2-Ubuntu <<>> novaproxy-01.project-proxy.eqiad.wmflabs @labs-ns0.wikimedia.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21651
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 2800
;; QUESTION SECTION:
;novaproxy-01.project-proxy.eqiad.wmflabs. IN A
;; ANSWER SECTION:
novaproxy-01.project-proxy.eqiad.wmflabs. 60 IN	A 10.68.21.68
;; Query time: 91 msec
;; SERVER: 208.80.155.117#53(208.80.155.117)
;; WHEN: Wed Nov 07 12:48:08 GMT 2018
;; MSG SIZE  rcvd: 85

The public DNS stuff lives under the wmflabsdotorg tenant:

krenair@shinken-02:~$ OS_PROJECT_ID=wmflabsdotorg openstack zone list
+--------------------------------------+---------------------------------+---------+------------+--------+--------+
| id                                   | name                            | type    |     serial | status | action |
+--------------------------------------+---------------------------------+---------+------------+--------+--------+
| 553ef162-add7-4a5c-b115-9cabca662746 | wmflabs.org.                    | PRIMARY | 1541584686 | ACTIVE | NONE   |
| ef825cf2-db2d-4480-ad33-2c19b0a188dc | wmflabsdotorg.wmflabs.org.      | PRIMARY | 1535835306 | ACTIVE | NONE   |
| 933a78c2-3d8d-4ee1-bbef-9ab30be5f972 | 128-25.155.80.208.in-addr.arpa. | PRIMARY | 1535835306 | ACTIVE | NONE   |
| e07defe3-6d08-4f37-bd7d-1bdc1c45d7e8 | 56.15.185.in-addr.arpa.         | PRIMARY | 1541514722 | ACTIVE | NONE   |
+--------------------------------------+---------------------------------+---------+------------+--------+--------+

When floating IPs are allocated and assigned, and DNS records pointed at them like so:

krenair@shinken-02:~$ OS_PROJECT_ID=wmflabsdotorg openstack recordset show wmflabs.org. ntp-01.wmflabs.org.
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| action      | NONE                                 |
| created_at  | 2018-11-02T20:34:18.000000           |
| description | time server for cloud instances      |
| id          | 5812d178-b7a8-4168-bce7-5e064b411f82 |
| name        | ntp-01.wmflabs.org.                  |
| records     | 185.15.56.3                          |
| status      | ACTIVE                               |
| ttl         | 3600                                 |
| type        | A                                    |
| updated_at  | None                                 |
| version     | 1                                    |
| zone_id     | 553ef162-add7-4a5c-b115-9cabca662746 |
+-------------+--------------------------------------+

Alex's dns-floating-ip-updater.py script creates something like this:

krenair@shinken-02:~$ OS_PROJECT_ID=wmflabsdotorg openstack recordset show 56.15.185.in-addr.arpa. 3.56.15.185.in-addr.arpa.
+-------------+---------------------------------------------------------------------------+
| Field       | Value                                                                     |
+-------------+---------------------------------------------------------------------------+
| action      | NONE                                                                      |
| created_at  | 2018-11-02T20:41:47.000000                                                |
| description | MANAGED BY dns-floating-ip-updater.py IN PUPPET - DO NOT UPDATE OR DELETE |
| id          | bad4939b-aaf3-4bef-8878-2351b8944b76                                      |
| name        | 3.56.15.185.in-addr.arpa.                                                 |
| records     | ntp-01.wmflabs.org.                                                       |
|             | ntp-01.cloudinfra.wmflabs.org.                                            |
| status      | ACTIVE                                                                    |
| ttl         | None                                                                      |
| type        | PTR                                                                       |
| updated_at  | 2018-11-02T21:01:45.000000                                                |
| version     | 2                                                                         |
| zone_id     | e07defe3-6d08-4f37-bd7d-1bdc1c45d7e8                                      |
+-------------+---------------------------------------------------------------------------+

and usually instance-$instance.$project.wmflabs.org records to make it obvious which IP is served by which host.

Designate services

Designate has a lot of moving parts. On a given designate hosts, you will (or may) see the following things running:

  • designate-api: the REST endpoint for creating or deleting zones or records. Horizon and the commandline client talk directly to this.
  • designate-sink: listens (via rabbitmq) for nova notifications about instance creation or deletion; creates and deletes records under .eqiad.wmflabs as needed.
  • designate-mdns: a tiny dns server maintained by designate -- this exists mainly as a source of authority for xxfr sync requests with powerdns. xfr can only sync records and not create domains.
  • designate-pool-manager: since xxfr is only useful for syncing records, designate-pool-manager holds credentials to write directly to the pdns database; it uses those creds to insert or remove pdns domain records as needed.
  • designate-central: coordinates all of the above
  • designate-agent: a leftover service from an older version of designate that doesn't do anything but is somehow still installed by the .deb packges. If it's running, feel free to stop it; puppet doesn't ensure that it's either up or down.

Note that this suite of services gets renamed and/or refactored with almost every Designate release. The above list is for Designate Mitaka; the list is almost certainly different in later releases.

Initial designate/pdns node setup

A lot of the node config is puppetized, but there are several steps that need manual intervention. This is kind of a mess, and more of it should probably be moved into puppet at some point.

Initial puppet runs will complain about the inability to start mariadb. This is complicated on Stretch by the mariadb package being a bit broken. To resolve:

 root@cloudservices1004:ln -s /opt/wmf-mariadb101 /opt/wmf-mariadb10
 root@cloudservices1004:/opt# cd /opt/wmf-mariadb10
 root@cloudservices1004:/opt/wmf-mariadb10# ./scripts/mysql_install_db
   Installing MariaDB/MySQL system tables in '/srv/sqldata' ...

Once puppet runs are clean, create the initial pdns database and set up grants. First you'll need to find the socket file:

 root@cloudservices1004:/tmp# find . -name "mysql*"
   ./systemd-private-2588877b841f451db29adcf4b35a2afa-mariadb.service-WEFqZB/tmp/mysql.sock
 root@cloudservices1004:/tmp# mysql --socket systemd-private-2588877b841f451db29adcf4b35a2afa-mariadb.service-WEFqZB/tmp/mysql.sock
 MariaDB [(none)]> create database pdns;
 Query OK, 1 row affected (0.00 sec)
 MariaDB [(none)]> USE pdns;
 Database changed
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'localhost' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.00 sec)
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'<first designate node ipv4>' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.01 sec)
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'<second designate node ipv4>' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.01 sec)
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'<first designate node ipv6>' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.01 sec)
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'<second designate node ipv4>' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.00 sec)

The pdns-recursor will expect some ip alias files to exist. Those files are maintained by a timer, but to get things running right now, run

#/usr/local/bin/labs-ip-alias-dump.py

Then restart pdns-recursor.

If adding a new node (as opposed to rebuilding an existing one with a previously registered IP), you'll need to update the designate pool to include the new node. A template of the yaml pool config can be found in puppet: puppet/modules/openstack/files/mitaka/designate/eqiad1_pool_config.yml. Copy it onto an active designate nodes, edit as needed, and then:

 designate-manage pool update  --file ./eqiad1_pool_config.yml --dry_run true

and if everything parses properly,

   designate-manage pool update  --file ./eqiad1_pool_config.yml

Finally, whether this is a newly added node or a rebuild, we need designate to set up the pdns database schema. Online pdns guides will tell you that the pdns schema needs to be set up by hand, but designate will do it for us, manage version tracking, and insert some extra fields that designate needs to keep track of what it's managing.

 # designate-manage pool show_config
 #  # note the pool id output from this
 # designate-manage powerdns upgrade <pool id from the pool config file above>
 # designate-manage powerdns sync <pool id from the pool config file above>

Now we're ready to restart all designate services (and pdns) and see what's broken. Rebooting is easiest!

Now pdns will probably be complaining (in the syslog) about getting axfr requests for unknown domains. That's because designate doesn't automatically populate domains in pdns, it only creates them as needed when records are added. We have too many domains to wait, so the best way forward is to dump the 'domains' table from an existing node and import it on the new node.

 #  # on existing, working node:
 # mysqldump pdns domains > domains.sql
 #  # on new node
 # mysql --socket <whatever> pdns < domains.sql

Finally: designate-sink needs to be able to talk to the nova proxy, in order to clean up proxies associated with a deleted VM. Add access to port 5668 for the IP of the new node to project-proxy's security group. Similarly, when creating a new node make sure it can talk to the cloud puppetmasters on port 8101 so it can clean up the certs associated with a deleted VM.

New PTR record

Example command to create a new A record:

root@cloudcontrol1003:~# designate --all-tenants --os-project-name admin record-create 16.172.in-addr.arpa. \
>  --name 1.0.16.172.in-addr.arpa. --type PTR --data cloudinstances2b-gw.svc.eqiad.wmflabs. \
>  --description "Neutron virtual router. Record created by hand"
+-------------+------------------------------------------------+
| Field       | Value                                          |
+-------------+------------------------------------------------+
| description | Neutron virtual router. Record created by hand |
| type        | PTR                                            |
| created_at  | 2018-11-30T13:26:47.000000                     |
| updated_at  | None                                           |
| domain_id   | 6990e139-49e6-466c-9421-46cf45f05842           |
| priority    | None                                           |
| ttl         | None                                           |
| data        | cloudinstances2b-gw.svc.eqiad.wmflabs.       |
| id          | 855a3d4d-4caf-4652-b897-d6559a64bb4e           |
| name        | 1.0.16.172.in-addr.arpa.                       |
+-------------+------------------------------------------------+

New A records

Example command of updating a PTR record:

root@cloudcontrol1003:~# designate --all-tenants --os-project-name admin record-create svc.eqiad.wmflabs. \
>  --name cloudinstances2b-gw.svc.eqiad.wmflabs. --type A --data 172.16.0.1 --description "Neutron virtual router. Record created by hand"
+-------------+------------------------------------------------+
| Field       | Value                                          |
+-------------+------------------------------------------------+
| description | Neutron virtual router. Record created by hand |
| type        | A                                              |
| created_at  | 2018-11-30T13:23:02.000000                     |
| updated_at  | None                                           |
| domain_id   | 114f1333-c2c1-44d3-beb4-ebed1a91742b           |
| priority    | None                                           |
| ttl         | None                                           |
| data        | 172.16.0.1                                     |
| id          | a92d47c2-aec3-4777-ac9f-cb246b3c9e13           |
| name        | cloudinstances2b-gw.svc.eqiad.wmflabs.         |
+-------------+------------------------------------------------+

New NS record

Example command for creating a new NS record by hand (sub domain delegation):

root@cloudcontrol1004:~# designate --all-tenants --os-project-name cloudinfra record-create wmcloud.org. --name codfw1dev.wmcloud.org --type NS --data ns0.openstack.codfw1dev.wikimediacloud.org --description "delegation to Designate in the other deployment. Record created by hand"
+-------------+-------------------------------------------------------------------------+
| Field       | Value                                                                   |
+-------------+-------------------------------------------------------------------------+
| description | delegation to Designate in the other deployment. Record created by hand |
| type        | NS                                                                      |
| created_at  | 2020-01-28T10:02:28.000000                                              |
| updated_at  | None                                                                    |
| domain_id   | a5f22422-815c-43d2-95aa-8fbdee95de01                                    |
| priority    | None                                                                    |
| ttl         | None                                                                    |
| data        | ns0.openstack.codfw1dev.wikimediacloud.org.                             |
| id          | a0a86d45-6bab-445f-8bd2-f0bf2f76e45a                                    |
| name        | codfw1dev.wmcloud.org.                                                  |
+-------------+-------------------------------------------------------------------------+

Delegations within the same openstack deployment are usually made using the wmcs-makedomain script.

Updating an existing record

Example command for updating an existing A record:

# Update A record:
# designate --all-tenants --os-project-name admin record-update --name tools.db.svc.eqiad.wmflabs. --type A --data 185.15.56.54 --description "ToolsDB server. Record created by hand" df88fcb3-fbc2-42f1-bb12-2424c8b7117e 522952da-b43b-4032-bcb5-df04b6a1cdbc
# which means (uids translated):
# designate --all-tenants --os-project-name admin record-update --name tools.db.svc.eqiad.wmflabs. --type A --data 185.15.56.54 --description "ToolsDB server. Record created by hand" db.svc.eqiad.wmflabs. tools.db.svc.eqiad.wmflabs.

TODO: verify me and use proper syntax highlighting.

Detecting leaked records

In some cases, leaked DNS records may happen. We have a custom script to detect/correct them: wmcs-novastats-dnsleaks. If this script is run with the --delete argument, it will delete leaked records, which is usefull if there are many of them.

root@cloudcontrol1003:~# wmcs-novastats-dnsleaks
A record for huggle-pg.huggle.eqiad.wmflabs. has multiple IPs: ['10.68.17.231', '172.16.2.31']
This needs cleanup but that isn't implemented and almost never happens.
[...]
a6cf149d-d90f-401c-a380-a83b65a69d79 is linked to missing instance cloudinstances2b-gw.admin.eqiad.wmflabs.
[...]
PTR e06fdc2d-2bcd-40ec-b416-ca4d63f0dce2 is linked to missing instance ci-jessie-wikimedia-1099005.contintcloud.eqiad.wmflab

tools.eqiad.wmflabs

Some special hosts are defined in profile::openstack::main::pdns::recursor_aliaser_extra_records:

  • tools-db.tools.eqiad.wmflabs
  • tools-redis.tools.eqiad.wmflabs
  • tools-redis.eqiad.wmflabs
  • puppet

DNS zone creating / transfers

According to our DNS setup, certain domains (zones in Designate) should be created in the admin project and then transfered / relocated to the cloudinfra project. This can be done with the openstack native transfer mechanism.

# openstack zone transfer request create --target-project-id cloudinfra 67603ef4-3d64-40d6-90d3-5b7776a99034
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                               |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| created_at        | 2020-02-18T15:35:43.000000                                                                                                          |
| description       | None                                                                                                                                |
| id                | 0a30d1fe-51ea-4da4-8a27-80c16e681ff6                                                                                                |
| key               | LOYRJY7Q                                                                                                                            |
| links             | {u'self': u'http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones/tasks/transfer_requests/0a30d1fe-51ea-4da4-8a27-80c16e681ff6'} |
| project_id        | admin                                                                                                                               |
| status            | ACTIVE                                                                                                                              |
| target_project_id | cloudinfra                                                                                                                          |
| updated_at        | None                                                                                                                                |
| zone_id           | 67603ef4-3d64-40d6-90d3-5b7776a99034                                                                                                |
| zone_name         | eqiad1.wikimedia.cloud.                                                                                                             |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+

# OS_PROJECT_ID=cloudinfra openstack zone transfer accept request --key LOYRJY7Q --transfer-id 0a30d1fe-51ea-4da4-8a27-80c16e681ff6 
+--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                    | Value                                                                                                                                                                                                                                     |
+--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at               | 2020-02-18T15:38:17.000000                                                                                                                                                                                                                |
| id                       | f8077a78-9f0d-4cbc-9ac7-59873f5657fa                                                                                                                                                                                                      |
| key                      | LOYRJY7Q                                                                                                                                                                                                                                  |
| links                    | {u'self': u'http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones/tasks/transfer_accepts/f8077a78-9f0d-4cbc-9ac7-59873f5657fa', u'zone': u'http://cloudservices1003.wikimedia.org:9001/v2/zones/67603ef4-3d64-40d6-90d3-5b7776a99034'} |
| project_id               | cloudinfra                                                                                                                                                                                                                                |
| status                   | COMPLETE                                                                                                                                                                                                                                  |
| updated_at               | 2020-02-18T15:38:17.000000                                                                                                                                                                                                                |
| zone_id                  | 67603ef4-3d64-40d6-90d3-5b7776a99034                                                                                                                                                                                                      |
| zone_transfer_request_id | 0a30d1fe-51ea-4da4-8a27-80c16e681ff6                                                                                                                                                                                                      |
+--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

# OS_PROJECT_ID=cloudinfra openstack zone show 67603ef4-3d64-40d6-90d3-5b7776a99034
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| action         | NONE                                 |
| attributes     | {}                                   |
| created_at     | 2019-10-18T15:57:51.000000           |
| description    | None                                 |
| email          | root@wmflabs.org                     |
| id             | 67603ef4-3d64-40d6-90d3-5b7776a99034 |
| masters        |                                      |
| name           | eqiad1.wikimedia.cloud.              |
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id     | cloudinfra                           |
| serial         | 1581676303                           |
| status         | ACTIVE                               |
| transferred_at | None                                 |
| ttl            | 3600                                 |
| type           | PRIMARY                              |
| updated_at     | 2020-02-18T15:38:17.000000           |
| version        | 6                                    |
+----------------+--------------------------------------+

See phabricator T245494 - CloudVPS: figure out DNS zone ownership transfers and setup for context on how this was figured out.

Administrative scripts

See CloudVPS maintenance, DNS admin scripts.

Wiki Replica DNS

See Portal:Data_Services/Admin/Wiki_Replica_DNS

labs-ip-alias-dump

This script actually generates a lua script which is read by pdns-recursor. This is not technical anything related to Designate, but it is relevant to Cloud VPS DNS lookups. The script hooks two lua resolver hooks:

postresolve
Maps public IPs to private IPs so that a lookup that results in a floating IP instead returns the private IP of the instance that it is attached to.
preresolve
Injects additional DNS results that are only visible via the recursor. Currently used to provide DNS responses for:
  • puppet.
  • tools-db.tools.eqiad.wmflabs.
  • tools-redis.eqiad.wmflabs.
  • tools-redis.tools.eqiad.wmflabs.

The script runs via cron every 30 minutes on each pdns-recursor host.

dns-floating-ip-updater

Labs has a block of public IPs: 208.80.155.128/25

Unfortunately 208.80.155.0/25 are production IPs

Therefore production can't delegate the whole of the 155.80.208.in-addr.arpa. zone to labs NS servers for us to automate reverse DNS records from

Our workaround is RFC 2317-style delegation: We have a zone called 128-25.155.80.208.in-addr.arpa. (living under the wmflabsdotorg tenant IIRC) that production CNAMEs all the usual in-addr records for labs into, and delegates the zone to labs nameservers.

We have a script called dns-floating-ip-updater that sets up instance-{instancename}.{project}.wmflabs.org records (assuming {project}.wmflabs.org exists as a domain, if not they don't get the feature) and the reverse DNS record for that, then also sets up the reverse DNS for all manually curated A records.

What happens when someone makes a reverse DNS lookup, for let's say 208.80.155.131:

  • It gets turned into a PTR lookup on 131.155.80.208.in-addr.arpa. as normal
  • Prod NS servers control the 155.80.208.in-addr.arpa. zone - if you open the file for that in the operations/dns.git repository ('templates' directory), you'll see the first 128 addresses are reserved for prod (some in use, some not)
  • So under 208.80.155, number 131 is IN CNAME 131.128-25 i.e. 131.128-25.155.80.208.in-addr.arpa.
  • 128-25.155.80.208.in-addr.arpa. is delegated from prod to labs NS, i.e. designate.
  • So the full lookup becomes 131.128-25.155.80.208.in-addr.arpa. and is sent to the labs NS server
  • This script ensures that designate knows how to answer that query, by populating all the records.

For example:

$ host bastion.wmflabs.org
bastion.wmflabs.org has address 208.80.155.129 # this is answered by the labs nameservers (designate) as they control wmflabs.org, under the wmflabsdotorg tenant
$ host 208.80.155.129
129.155.80.208.in-addr.arpa is an alias for 129.128-25.155.80.208.in-addr.arpa. # this is the production nameservers delegating the query to labs designate
129.128-25.155.80.208.in-addr.arpa domain name pointer bastion.wmflabs.org. # this is from the labs nameservers (designate) and is because bastion.wmflabs.org points to that IP
129.128-25.155.80.208.in-addr.arpa domain name pointer instance-bastion-01.bastion.wmflabs.org. # this comes from the labs nameservers (designate) and is used to indicate which instance a given IP points to, partially in case it lacks other names, but it's helpful regardless. (think what happens if we didn't have this entry but you wanted to go look up the instance on the project's instance list knowing only it's IP - and that's assuming you know which project to look under)
129.128-25.155.80.208.in-addr.arpa domain name pointer primary.bastion.wmflabs.org. # this is from the labs nameservers (designate) and is because primary.bastion.wmflabs.org points to that IP
$ host instance-bastion-01.bastion.wmflabs.org
instance-bastion-01.bastion.wmflabs.org has address 208.80.155.129 # this comes from the labs nameservers (designate) - wouldn't make sense to provide the PTR above without this

We will need to do some work on our script to be able to add new domains or public IP ranges (e.g. for other datacentres, to introduce IPv6, or to simply get more addresses).

Announcement post: https://www.mail-archive.com/labs-l@lists.wikimedia.org/msg04516.html