Portal:Cloud VPS/Admin/DNS/Designate

From Wikitech
Jump to navigation Jump to search

Cloud VPS DNS is PowerDNS, backed by a database controlled by OpenStack Designate.

Basic info

When an instance is created a DNS A record is created in Designate.

$ dig +noall +answer tools-sgebastion-08.tools.eqiad1.wikimedia.cloud
tools-sgebastion-08.tools.eqiad1.wikimedia.cloud. 3541 IN A 172.16.3.190

Root zones for public facing DNS records are owned by either the cloudinfra or wmflabsdotorg tenants:

$ sudo wmcs-openstack --os-project-id=cloudinfra zone list --max-width=80
+------------------+------------------+---------+------------+--------+--------+
| id               | name             | type    |     serial | status | action |
+------------------+------------------+---------+------------+--------+--------+
| d863546e-cde7-4a | cloudinfra.wmfla | PRIMARY | 1583906708 | ACTIVE | NONE   |
| 4c-b2e2-a7e89db0 | bs.org.          |         |            |        |        |
| f196             |                  |         |            |        |        |
| 67603ef4-3d64-40 | eqiad1.wikimedia | PRIMARY | 1589574668 | ACTIVE | NONE   |
| d6-90d3-5b7776a9 | .cloud.          |         |            |        |        |
| 9034             |                  |         |            |        |        |
| 6ddcb082-69d6-43 | wmcloud.org.     | PRIMARY | 1585587288 | ACTIVE | NONE   |
| f4-9993-5c6bdc27 |                  |         |            |        |        |
| dfc9             |                  |         |            |        |        |
+------------------+------------------+---------+------------+--------+--------+
$ sudo wmcs-openstack --os-project-id=wmflabsdotorg zone list --max-width=80
+------------------+------------------+---------+------------+--------+--------+
| id               | name             | type    |     serial | status | action |
+------------------+------------------+---------+------------+--------+--------+
| 553ef162-add7-4a | wmflabs.org.     | PRIMARY | 1589567523 | ACTIVE | NONE   |
| 5c-b115-9cabca66 |                  |         |            |        |        |
| 2746             |                  |         |            |        |        |
| ef825cf2-db2d-44 | wmflabsdotorg.wm | PRIMARY | 1583906762 | ACTIVE | NONE   |
| 80-ad33-2c19b0a1 | flabs.org.       |         |            |        |        |
| 88dc             |                  |         |            |        |        |
| 933a78c2-3d8d-4e | 128-25.155.80.20 | PRIMARY | 1583906762 | ACTIVE | NONE   |
| e1-bbef-9ab30be5 | 8.in-addr.arpa.  |         |            |        |        |
| f972             |                  |         |            |        |        |
| e07defe3-6d08-4f | 56.15.185.in-    | PRIMARY | 1589491014 | ACTIVE | NONE   |
| 37-bd7d-1bdc1c45 | addr.arpa.       |         |            |        |        |
| d7e8             |                  |         |            |        |        |
+------------------+------------------+---------+------------+--------+--------+

Reverse zones (and some legacy internal zones) are owned by the special noauth-project tenant which novaobserver cannot access.

$ sudo wmcs-openstack zone list --sudo-project-id noauth-project --fit-width
+------------------+------------------+---------+------------+--------+--------+
| id               | name             | type    |     serial | status | action |
+------------------+------------------+---------+------------+--------+--------+
| 114f1333-c2c1-44 | eqiad.wmflabs.   | PRIMARY | 1589574227 | ACTIVE | NONE   |
| d3-beb4-ebed1a91 |                  |         |            |        |        |
| 742b             |                  |         |            |        |        |
| 8d114f3c-815b-46 | 68.10.in-        | PRIMARY | 1582041663 | ACTIVE | NONE   |
| 6c-bdd4-9b91f704 | addr.arpa.       |         |            |        |        |
| ea60             |                  |         |            |        |        |
| d19aff2d-2d57-4d | svc.eqiad.wmflab | PRIMARY | 1585953047 | ACTIVE | NONE   |
| 25-9e26-dab2b3a5 | s.               |         |            |        |        |
| 8be4             |                  |         |            |        |        |
| df88fcb3-fbc2-42 | db.svc.eqiad.wmf | PRIMARY | 1583876852 | ACTIVE | NONE   |
| f1-bb12-2424c8b7 | labs.            |         |            |        |        |
| 117e             |                  |         |            |        |        |
| e81ea3bb-f8f0-49 | web.db.svc.eqiad | PRIMARY | 1586976118 | ACTIVE | NONE   |
| f4-906b-2d4c2e83 | .wmflabs.        |         |            |        |        |
| cc7e             |                  |         |            |        |        |
| 04c45c1f-214d-45 | analytics.db.svc | PRIMARY | 1586975792 | ACTIVE | NONE   |
| 0b-a733-028dcdc8 | .eqiad.wmflabs.  |         |            |        |        |
| 7a12             |                  |         |            |        |        |
| 6990e139-49e6-46 | 16.172.in-       | PRIMARY | 1589574225 | ACTIVE | NONE   |
| 6c-9421-46cf45f0 | addr.arpa.       |         |            |        |        |
| 5842             |                  |         |            |        |        |
+------------------+------------------+---------+------------+--------+--------+


When floating IPs are allocated and assigned, DNS records can be created for them using Horizon:

$ sudo wmcs-openstack --os-project-id=wmflabsdotorg recordset show wmflabs.org. ntp-01.wmflabs.org.
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| action      | NONE                                 |
| created_at  | 2018-11-02T20:34:18.000000           |
| description | time server for cloud instances      |
| id          | 5812d178-b7a8-4168-bce7-5e064b411f82 |
| name        | ntp-01.wmflabs.org.                  |
| project_id  | wmflabsdotorg                        |
| records     | 185.15.56.3                          |
| status      | ACTIVE                               |
| ttl         | 3600                                 |
| type        | A                                    |
| updated_at  | None                                 |
| version     | 1                                    |
| zone_id     | 553ef162-add7-4a5c-b115-9cabca662746 |
| zone_name   | wmflabs.org.                         |
+-------------+--------------------------------------+

The dns-floating-ip-updater script runs as a systemd timer to ensure that A and PTR records are created for each floating IP. The automatic A record will be of the form instance-$instance.$project.wmflabs.org to make it obvious which IP is served by which instance.

$ sudo wmcs-openstack --os-project-id=wmflabsdotorg recordset show 56.15.185.in-addr.arpa. 3.56.15.185.in-addr.arpa. --max-width=80
+-------------+----------------------------------------------------------------+
| Field       | Value                                                          |
+-------------+----------------------------------------------------------------+
| action      | NONE                                                           |
| created_at  | 2018-12-17T19:12:11.000000                                     |
| description | MANAGED BY dns-floating-ip-updater.py IN PUPPET - DO NOT       |
|             | UPDATE OR DELETE                                               |
| id          | 5d331e2b-53d1-407e-bf65-ad022ac8e186                           |
| name        | 3.56.15.185.in-addr.arpa.                                      |
| project_id  | wmflabsdotorg                                                  |
| records     | ntp-01.cloudinfra.wmflabs.org.                                 |
|             | ntp-01.wmflabs.org.                                            |
|             | instance-ntp-01.cloudinfra.wmflabs.org.                        |
| status      | ACTIVE                                                         |
| ttl         | None                                                           |
| type        | PTR                                                            |
| updated_at  | 2019-12-12T07:14:47.000000                                     |
| version     | 4                                                              |
| zone_id     | e07defe3-6d08-4f37-bd7d-1bdc1c45d7e8                           |
| zone_name   | 56.15.185.in-addr.arpa.                                        |
+-------------+----------------------------------------------------------------+

Designate services

Designate has a lot of moving parts. On a given designate hosts, you will (or may) see the following things running:

  • designate-api: the REST endpoint for creating or deleting zones or records. Horizon and the commandline client talk directly to this.
  • designate-sink: listens (via rabbitmq) for nova notifications about instance creation or deletion; creates and deletes records under .eqiad.wmflabs as needed.
  • designate-mdns: a tiny dns server maintained by designate -- this exists mainly as a source of authority for xxfr sync requests with powerdns. xfr can only sync records and not create domains.
  • designate-pool-manager: since xxfr is only useful for syncing records, designate-pool-manager holds credentials to write directly to the pdns database; it uses those creds to insert or remove pdns domain records as needed.
  • designate-central: coordinates all of the above
  • designate-agent: a leftover service from an older version of designate that doesn't do anything but is somehow still installed by the .deb packges. If it's running, feel free to stop it; puppet doesn't ensure that it's either up or down.

Note that this suite of services gets renamed and/or refactored with almost every Designate release. The above list is for Designate Mitaka; the list is almost certainly different in later releases.

Initial designate/pdns node setup

A lot of the node config is puppetized, but there are several steps that need manual intervention. This is kind of a mess, and more of it should probably be moved into puppet at some point.

Initial puppet runs will complain about the inability to start mariadb. This is complicated on Stretch by the mariadb package being a bit broken. To resolve:

 root@cloudservices1004:ln -s /opt/wmf-mariadb101 /opt/wmf-mariadb10
 root@cloudservices1004:/opt# cd /opt/wmf-mariadb10
 root@cloudservices1004:/opt/wmf-mariadb10# ./scripts/mysql_install_db
   Installing MariaDB/MySQL system tables in '/srv/sqldata' ...

Once puppet runs are clean, create the initial pdns database and set up grants. First you'll need to find the socket file:

 root@cloudservices1004:/tmp# find . -name "mysql*"
   ./systemd-private-2588877b841f451db29adcf4b35a2afa-mariadb.service-WEFqZB/tmp/mysql.sock
 root@cloudservices1004:/tmp# mysql --socket systemd-private-2588877b841f451db29adcf4b35a2afa-mariadb.service-WEFqZB/tmp/mysql.sock
 MariaDB [(none)]> create database pdns;
 Query OK, 1 row affected (0.00 sec)
 MariaDB [(none)]> USE pdns;
 Database changed
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'localhost' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.00 sec)
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'<first designate node ipv4>' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.01 sec)
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'<second designate node ipv4>' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.01 sec)
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'<first designate node ipv6>' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.01 sec)
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'<second designate node ipv4>' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.00 sec)

The pdns-recursor will expect some ip alias files to exist. Those files are maintained by a timer, but to get things running right now, run

#/usr/local/bin/labs-ip-alias-dump.py

Then restart pdns-recursor.

If adding a new node (as opposed to rebuilding an existing one with a previously registered IP), you'll need to update the designate pool to include the new node. A template of the yaml pool config can be found in puppet: puppet/modules/openstack/files/mitaka/designate/eqiad1_pool_config.yml. Copy it onto an active designate nodes, edit as needed, and then:

 designate-manage pool update  --file ./eqiad1_pool_config.yml --dry_run true

and if everything parses properly,

   designate-manage pool update  --file ./eqiad1_pool_config.yml

If adding a new node you will also need to update pdns on all nodes so everyone agrees about who is managing the domains:

 update domains set master="<designate node 1 ipv4>:5354 <designate node 2 ipv4>:5354";

Finally, whether this is a newly added node or a rebuild, we need designate to set up the pdns database schema. Online pdns guides will tell you that the pdns schema needs to be set up by hand, but designate will do it for us, manage version tracking, and insert some extra fields that designate needs to keep track of what it's managing.

 # designate-manage pool show_config
 #  # note the pool id output from this
 # designate-manage powerdns upgrade <pool id from the pool config file above>
 # designate-manage powerdns sync <pool id from the pool config file above>

Now we're ready to restart all designate services (and pdns) and see what's broken. Rebooting is easiest!

Now pdns will probably be complaining (in the syslog) about getting axfr requests for unknown domains. That's because designate doesn't automatically populate domains in pdns, it only creates them as needed when records are added. We have too many domains to wait, so the best way forward is to dump the 'domains' table from an existing node and import it on the new node.

 #  # on existing, working node:
 # mysqldump pdns domains > domains.sql
 #  # on new node
 # mysql --socket <whatever> pdns < domains.sql

Finally: designate-sink needs to be able to talk to the nova proxy, in order to clean up proxies associated with a deleted VM. Add access to port 5668 for the IP of the new node to project-proxy's security group. Similarly, when creating a new node make sure it can talk to the cloud puppetmasters on port 8101 so it can clean up the certs associated with a deleted VM.

New PTR record

Example command to create a new PTR record:

# openstack --os-project-id cloudinfra-codfw1dev recordset create --type PTR --record test.codfw1dev.wmcloud.org. --description "test" 0-29.57.15.185.in-addr.arpa. 1.0-29.57.15.185.in-addr.arpa.
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| action      | CREATE                               |
| created_at  | 2020-05-25T17:07:34.000000           |
| description | test                                 |
| id          | 450a83c9-cca2-4c82-8a28-13cf035c7702 |
| name        | 1.0-29.57.15.185.in-addr.arpa.       |
| project_id  | cloudinfra-codfw1dev                 |
| records     | test.codfw1dev.wmcloud.org.          |
| status      | PENDING                              |
| ttl         | None                                 |
| type        | PTR                                  |
| updated_at  | None                                 |
| version     | 1                                    |
| zone_id     | 7796b60e-7bc6-4cb4-9e4d-dae039d3f912 |
| zone_name   | 0-29.57.15.185.in-addr.arpa.         |
+-------------+--------------------------------------+

In previous openstack versions we were using the designate CLI, but as of Rocky this probably no longer works:

root@cloudcontrol1003:~# designate --all-tenants --os-project-name admin record-create 16.172.in-addr.arpa. \
>  --name 1.0.16.172.in-addr.arpa. --type PTR --data cloudinstances2b-gw.svc.eqiad.wmflabs. \
>  --description "Neutron virtual router. Record created by hand"
+-------------+------------------------------------------------+
| Field       | Value                                          |
+-------------+------------------------------------------------+
| description | Neutron virtual router. Record created by hand |
| type        | PTR                                            |
| created_at  | 2018-11-30T13:26:47.000000                     |
| updated_at  | None                                           |
| domain_id   | 6990e139-49e6-466c-9421-46cf45f05842           |
| priority    | None                                           |
| ttl         | None                                           |
| data        | cloudinstances2b-gw.svc.eqiad.wmflabs.       |
| id          | 855a3d4d-4caf-4652-b897-d6559a64bb4e           |
| name        | 1.0.16.172.in-addr.arpa.                       |
+-------------+------------------------------------------------+

New A records

Example command to create a new A record:

root@cloudcontrol1003:~# OS_PROJECT_ID=tools openstack recordset create --type A --records 172.16.0.24 --sudo-project-id tools svc.tools.eqiad1.wikimedia.cloud. elasticsearch
+-------------+-------------------------------------------------+
| Field       | Value                                           |   
+-------------+-------------------------------------------------+
| action      | CREATE                                          |   
| created_at  | 2020-03-03T18:15:41.000000                      |   
| description | None                                            |   
| id          | 35d2a87c-d355-4805-90d9-4a088f5981bc            |   
| name        | elasticsearch.svc.tools.eqiad1.wikimedia.cloud. |
| project_id  | tools                                           |   
| records     | 172.16.0.24                                     |   
| status      | PENDING                                         |   
| ttl         | None                                            |   
| type        | A                                               |   
| updated_at  | None                                            |   
| version     | 1                                               |   
| zone_id     | 808263bd-522d-42c3-88a1-40af24b2929e            |   
| zone_name   | svc.tools.eqiad1.wikimedia.cloud.               |   
+-------------+-------------------------------------------------+

New NS record

Example command for creating a new NS record by hand (sub domain delegation):

root@cloudcontrol1004:~# designate --all-tenants --os-project-name cloudinfra record-create wmcloud.org. --name codfw1dev.wmcloud.org --type NS --data ns0.openstack.codfw1dev.wikimediacloud.org --description "delegation to Designate in the other deployment. Record created by hand"
+-------------+-------------------------------------------------------------------------+
| Field       | Value                                                                   |
+-------------+-------------------------------------------------------------------------+
| description | delegation to Designate in the other deployment. Record created by hand |
| type        | NS                                                                      |
| created_at  | 2020-01-28T10:02:28.000000                                              |
| updated_at  | None                                                                    |
| domain_id   | a5f22422-815c-43d2-95aa-8fbdee95de01                                    |
| priority    | None                                                                    |
| ttl         | None                                                                    |
| data        | ns0.openstack.codfw1dev.wikimediacloud.org.                             |
| id          | a0a86d45-6bab-445f-8bd2-f0bf2f76e45a                                    |
| name        | codfw1dev.wmcloud.org.                                                  |
+-------------+-------------------------------------------------------------------------+

Delegations within the same openstack deployment are usually made using the wmcs-makedomain script.

Updating an existing record

Example command for updating an existing A record:

# Update A record:
# designate --all-tenants --os-project-name admin record-update --name tools.db.svc.eqiad.wmflabs. --type A --data 185.15.56.54 --description "ToolsDB server. Record created by hand" df88fcb3-fbc2-42f1-bb12-2424c8b7117e 522952da-b43b-4032-bcb5-df04b6a1cdbc
# which means (uids translated):
# designate --all-tenants --os-project-name admin record-update --name tools.db.svc.eqiad.wmflabs. --type A --data 185.15.56.54 --description "ToolsDB server. Record created by hand" db.svc.eqiad.wmflabs. tools.db.svc.eqiad.wmflabs.

TODO: verify me and use proper syntax highlighting.

Detecting leaked records

In some cases, leaked DNS records may happen. We have a custom script to detect/correct them: wmcs-novastats-dnsleaks. If this script is run with the --delete argument, it will delete leaked records, which is usefull if there are many of them.

root@cloudcontrol1003:~# wmcs-novastats-dnsleaks
A record for huggle-pg.huggle.eqiad.wmflabs. has multiple IPs: ['10.68.17.231', '172.16.2.31']
This needs cleanup but that isn't implemented and almost never happens.
[...]
a6cf149d-d90f-401c-a380-a83b65a69d79 is linked to missing instance cloudinstances2b-gw.admin.eqiad.wmflabs.
[...]
PTR e06fdc2d-2bcd-40ec-b416-ca4d63f0dce2 is linked to missing instance ci-jessie-wikimedia-1099005.contintcloud.eqiad.wmflab

tools.eqiad.wmflabs

Some special hosts are defined in profile::openstack::main::pdns::recursor_aliaser_extra_records:

  • tools-db.tools.eqiad.wmflabs
  • tools-redis.tools.eqiad.wmflabs
  • tools-redis.eqiad.wmflabs
  • puppet

DNS zone creating / transfers

If the zone you want to create doesn't have a parent zone already registered in Designate (i.e, is not a sub-zone), you can create the zone directly in the project it will live on:

# openstack --os-project-id cloudinfra-codfw1dev zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description "floating IPs subnet" 0-29.57.15.185.in-addr.arpa.
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| action         | CREATE                               |
| attributes     |                                      |
| created_at     | 2020-05-25T16:34:38.000000           |
| description    | floating IPs subnet                  |
| email          | root@wmflabs.org                     |
| id             | 7796b60e-7bc6-4cb4-9e4d-dae039d3f912 |
| masters        |                                      |
| name           | 0-29.57.15.185.in-addr.arpa.         |
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id     | cloudinfra-codfw1dev                 |
| serial         | 1590424478                           |
| status         | PENDING                              |
| transferred_at | None                                 |
| ttl            | 3600                                 |
| type           | PRIMARY                              |
| updated_at     | None                                 |
| version        | 1                                    |
+----------------+--------------------------------------+

But anyway, according to our DNS setup, certain domains (zones in Designate) should be created in the admin project and then transfered / relocated to the cloudinfra project.

This can be done with the openstack native transfer mechanism.

# openstack zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description "floating IPs subnet" 57.15.185.in-addr.arpa.
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| action         | CREATE                               |
| attributes     |                                      |
| created_at     | 2020-04-06T17:38:44.000000           |
| description    | floating IPs subnet                  |
| email          | root@wmflabs.org                     |
| id             | 24e82029-c970-4422-875f-e81d9e59e33a |
| masters        |                                      |
| name           | 57.15.185.in-addr.arpa.              |
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id     | admin                                |
| serial         | 1586194724                           |
| status         | PENDING                              |
| transferred_at | None                                 |
| ttl            | 3600                                 |
| type           | PRIMARY                              |
| updated_at     | None                                 |
| version        | 1                                    |
+----------------+--------------------------------------+

# openstack zone transfer request create --target-project-id cloudinfra 67603ef4-3d64-40d6-90d3-5b7776a99034
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                               |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| created_at        | 2020-02-18T15:35:43.000000                                                                                                          |
| description       | None                                                                                                                                |
| id                | 0a30d1fe-51ea-4da4-8a27-80c16e681ff6                                                                                                |
| key               | LOYRJY7Q                                                                                                                            |
| links             | {u'self': u'http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones/tasks/transfer_requests/0a30d1fe-51ea-4da4-8a27-80c16e681ff6'} |
| project_id        | admin                                                                                                                               |
| status            | ACTIVE                                                                                                                              |
| target_project_id | cloudinfra                                                                                                                          |
| updated_at        | None                                                                                                                                |
| zone_id           | 67603ef4-3d64-40d6-90d3-5b7776a99034                                                                                                |
| zone_name         | eqiad1.wikimedia.cloud.                                                                                                             |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+

# OS_PROJECT_ID=cloudinfra openstack zone transfer accept request --key LOYRJY7Q --transfer-id 0a30d1fe-51ea-4da4-8a27-80c16e681ff6 
[..]

# OS_PROJECT_ID=cloudinfra openstack zone show 67603ef4-3d64-40d6-90d3-5b7776a99034
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| action         | NONE                                 |
| attributes     | {}                                   |
| created_at     | 2019-10-18T15:57:51.000000           |
| description    | None                                 |
| email          | root@wmflabs.org                     |
| id             | 67603ef4-3d64-40d6-90d3-5b7776a99034 |
| masters        |                                      |
| name           | eqiad1.wikimedia.cloud.              |
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id     | cloudinfra                           |
| serial         | 1581676303                           |
| status         | ACTIVE                               |
| transferred_at | None                                 |
| ttl            | 3600                                 |
| type           | PRIMARY                              |
| updated_at     | 2020-02-18T15:38:17.000000           |
| version        | 6                                    |
+----------------+--------------------------------------+

See phabricator T245494 - CloudVPS: figure out DNS zone ownership transfers and setup for context on how this was figured out.

.wmflabs and .wikimedia.cloud

Records created by designate-sink are marked as 'managed' which means they can not be edited with Horizon or normal command line actions. This can be overridden with the --edit-managed flag, for example:

# OS_PROJECT_ID=cloudinfra openstack recordset set eqiad1.wikimedia.cloud. tools-redis-1003.tools.eqiad1.wikimedia.cloud. --record 172.16.1.107 --edit-managed

Records under .wmflabs (and a few other things including the ptr zone for internal IPs, 16.172.in-addr.arpa) are in a special visible-only-to-Designate tenant, 'noauth-project'. Horizon and many command-line tools are unable to access that tenant, but you can see them with the 'novaadmin' user and the --sudo-project-id flag:

# openstack zone list --sudo-project-id noauth-project

Records under .wikimedia.cloud are owned by the 'cloudinfra' project and can be viewed there. They are also managed, though, so modifying them requires use of the commandline and --edit-managed, as above.

svc zone in .wikimedia.cloud domain

To create and delegate the svc.NEWPROJECT.eqiad1.wikimedia.cloud domain, do the following. Please note that the parent domain always belong to the cloudinfra project per our setup.

  • Create the DNS zone for newproject in the cloudinfra project:
user@cloudcontrol1004:~ $ sudo wmcs-openstack --os-project-id cloudinfra zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 svc.newproject.eqiad1.wikimedia.cloud.
+----------------+---------------------------------------+
| Field          | Value                                 |
+----------------+---------------------------------------+
| action         | CREATE                                |
| attributes     |                                       |
| created_at     | 2020-05-21T12:27:05.000000            |
| description    | None                                  |
| email          | root@wmflabs.org                      |
| id             | 345c1fbc-4f15-4ab5-b87e-38bd284b55b9  |
| masters        |                                       |
| name           | svc.newproject.eqiad1.wikimedia.cloud.|
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842  |
| project_id     | cloudinfra                            |
| serial         | 1590064025                            |
| status         | PENDING                               |
| transferred_at | None                                  |
| ttl            | 3600                                  |
| type           | PRIMARY                               |
| updated_at     | None                                  |
| version        | 1                                     |
+----------------+---------------------------------------+
  • create the zone transfer request, mind the project specifications:
user@cloudcontrol1004:~ $ sudo wmcs-openstack --os-project-id cloudinfra zone transfer request create --target-project-id newproject 345c1fbc-4f15-4ab5-b87e-38bd284b55b9
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                             |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| created_at        | 2020-05-21T12:28:39.000000                                                                                                        |
| description       | None                                                                                                                              |
| id                | 3910d76b-b660-42a4-b995-21db53e3e42b                                                                                              |
| key               | KQBAHXJ4                                                                                                                          |
| links             | {'self': 'http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones/tasks/transfer_requests/3910d76b-b660-42a4-b995-21db53e3e42b'} |
| project_id        | cloudinfra                                                                                                                        |
| status            | ACTIVE                                                                                                                            |
| target_project_id | paws                                                                                                                              |
| updated_at        | None                                                                                                                              |
| zone_id           | 345c1fbc-4f15-4ab5-b87e-38bd284b55b9                                                                                              |
| zone_name         | svc.newproject.eqiad1.wikimedia.cloud.                                                                                            |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------+
  • accept the zone transfer request, again mind the project specifications:
user@cloudcontrol1004:~ $ sudo wmcs-openstack --os-project-id newproject zone transfer accept request --key G3VR0YQR --transfer-id 5d3367a7-2d3c-4ed7-8e44-8107ee9d7520
[...]
  • done! confirm the new ownership:
user@cloudcontrol1004:~ $ sudo wmcs-openstack zone show 345c1fbc-4f15-4ab5-b87e-38bd284b55b9 --os-project-id paws
+----------------+----------------------------------------+
| Field          | Value                                  |
+----------------+----------------------------------------+
| action         | NONE                                   |
| attributes     |                                        |
| created_at     | 2020-05-21T12:27:05.000000             |
| description    | None                                   |
| email          | root@wmflabs.org                       |
| id             | 345c1fbc-4f15-4ab5-b87e-38bd284b55b9   |
| masters        |                                        |
| name           | svc.newproject.eqiad1.wikimedia.cloud. |
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842   |
| project_id     | newproject                             |
| serial         | 1590064025                             |
| status         | ACTIVE                                 |
| transferred_at | None                                   |
| ttl            | 3600                                   |
| type           | PRIMARY                                |
| updated_at     | 2020-05-21T12:32:45.000000             |
| version        | 4                                      |
+----------------+----------------------------------------+

The instructions here are specific to this use case, including details on the ownership dancing, rather than the generic instructions in the DNS zone creating / transfers section.

Administrative scripts

See CloudVPS maintenance, DNS admin scripts.

Wiki Replica DNS

See Portal:Data_Services/Admin/Wiki_Replica_DNS

labs-ip-alias-dump

This script actually generates a lua script which is read by pdns-recursor. This is not technical anything related to Designate, but it is relevant to Cloud VPS DNS lookups. The script hooks two lua resolver hooks:

postresolve
Maps public IPs to private IPs so that a lookup that results in a floating IP instead returns the private IP of the instance that it is attached to.
preresolve
Injects additional DNS results that are only visible via the recursor. Currently used to provide DNS responses for:
  • puppet.
  • tools-db.tools.eqiad.wmflabs.
  • tools-redis.eqiad.wmflabs.
  • tools-redis.tools.eqiad.wmflabs.

The script runs via cron every 30 minutes on each pdns-recursor host.

dns-floating-ip-updater

Labs has a block of public IPs: 208.80.155.128/25

Unfortunately 208.80.155.0/25 are production IPs

Therefore production can't delegate the whole of the 155.80.208.in-addr.arpa. zone to labs NS servers for us to automate reverse DNS records from

Our workaround is RFC 2317-style delegation: We have a zone called 128-25.155.80.208.in-addr.arpa. (living under the wmflabsdotorg tenant IIRC) that production CNAMEs all the usual in-addr records for labs into, and delegates the zone to labs nameservers.

We have a script called dns-floating-ip-updater that sets up instance-{instancename}.{project}.wmflabs.org records (assuming {project}.wmflabs.org exists as a domain, if not they don't get the feature) and the reverse DNS record for that, then also sets up the reverse DNS for all manually curated A records.

What happens when someone makes a reverse DNS lookup, for let's say 208.80.155.131:

  • It gets turned into a PTR lookup on 131.155.80.208.in-addr.arpa. as normal
  • Prod NS servers control the 155.80.208.in-addr.arpa. zone - if you open the file for that in the operations/dns.git repository ('templates' directory), you'll see the first 128 addresses are reserved for prod (some in use, some not)
  • So under 208.80.155, number 131 is IN CNAME 131.128-25 i.e. 131.128-25.155.80.208.in-addr.arpa.
  • 128-25.155.80.208.in-addr.arpa. is delegated from prod to labs NS, i.e. designate.
  • So the full lookup becomes 131.128-25.155.80.208.in-addr.arpa. and is sent to the labs NS server
  • This script ensures that designate knows how to answer that query, by populating all the records.

For example:

$ host bastion.wmflabs.org
bastion.wmflabs.org has address 208.80.155.129 # this is answered by the labs nameservers (designate) as they control wmflabs.org, under the wmflabsdotorg tenant
$ host 208.80.155.129
129.155.80.208.in-addr.arpa is an alias for 129.128-25.155.80.208.in-addr.arpa. # this is the production nameservers delegating the query to labs designate
129.128-25.155.80.208.in-addr.arpa domain name pointer bastion.wmflabs.org. # this is from the labs nameservers (designate) and is because bastion.wmflabs.org points to that IP
129.128-25.155.80.208.in-addr.arpa domain name pointer instance-bastion-01.bastion.wmflabs.org. # this comes from the labs nameservers (designate) and is used to indicate which instance a given IP points to, partially in case it lacks other names, but it's helpful regardless. (think what happens if we didn't have this entry but you wanted to go look up the instance on the project's instance list knowing only it's IP - and that's assuming you know which project to look under)
129.128-25.155.80.208.in-addr.arpa domain name pointer primary.bastion.wmflabs.org. # this is from the labs nameservers (designate) and is because primary.bastion.wmflabs.org points to that IP
$ host instance-bastion-01.bastion.wmflabs.org
instance-bastion-01.bastion.wmflabs.org has address 208.80.155.129 # this comes from the labs nameservers (designate) - wouldn't make sense to provide the PTR above without this

We will need to do some work on our script to be able to add new domains or public IP ranges (e.g. for other datacentres, to introduce IPv6, or to simply get more addresses).

Announcement post: https://www.mail-archive.com/labs-l@lists.wikimedia.org/msg04516.html