Jump to content

Portal:Cloud VPS/Admin/DNS/Designate

From Wikitech

Cloud VPS DNS is PowerDNS, backed by a database controlled by OpenStack Designate. Designate services run on cloudcontrol node and use the same galera cluster database as other openstack services. PowerDNS runs on cloudservices boxes, and each instance has its own local mariadb database.

Designate DNS in WMCS

Basic info

When an instance is created a DNS A record is created in Designate. 

$ dig +noall +answer tools-sgebastion-08.tools.eqiad1.wikimedia.cloud
tools-sgebastion-08.tools.eqiad1.wikimedia.cloud. 3541 IN A 172.16.3.190

Root zones for public facing DNS records are owned by either the cloudinfra or wmflabsdotorg tenants: 

$ sudo wmcs-openstack --os-project-id=cloudinfra zone list --max-width=80
+--------------------------------------+----------------------------------------+---------+------------+--------+--------+
| id                                   | name                                   | type    |     serial | status | action |
+--------------------------------------+----------------------------------------+---------+------------+--------+--------+
| df88fcb3-fbc2-42f1-bb12-2424c8b7117e | db.svc.eqiad.wmflabs.                  | PRIMARY | 1732545689 | ACTIVE | NONE   |
| d863546e-cde7-4a4c-b2e2-a7e89db0f196 | cloudinfra.wmflabs.org.                | PRIMARY | 1727442153 | ACTIVE | NONE   |
| 67603ef4-3d64-40d6-90d3-5b7776a99034 | eqiad1.wikimedia.cloud.                | PRIMARY | 1732885287 | ACTIVE | NONE   |
| 6ddcb082-69d6-43f4-9993-5c6bdc27dfc9 | wmcloud.org.                           | PRIMARY | 1732881855 | ACTIVE | NONE   |
| 9e9a1892-a356-4f7e-8d47-a2bdb3aeb2f9 | svc.wikimedia.cloud.                   | PRIMARY | 1727442157 | ACTIVE | NONE   |
| d08cfa56-e646-4a1e-926c-003a34751dd3 | db.svc.wikimedia.cloud.                | PRIMARY | 1732545689 | ACTIVE | NONE   |
| 524eb3a1-fa1a-42a8-a166-47f419bfb107 | analytics.db.svc.wikimedia.cloud.      | PRIMARY | 1730310977 | ACTIVE | NONE   |
| 55dc12de-d948-4c52-9ebb-f90705b10864 | web.db.svc.wikimedia.cloud.            | PRIMARY | 1730311058 | ACTIVE | NONE   |
| 61c9b8ee-6392-43c0-8f94-f55a3032a5c3 | cloudinfra.wmcloud.org.                | PRIMARY | 1732050011 | ACTIVE | NONE   |
| 755b04d7-758e-4835-982a-0b99626e762e | svc.cloudinfra.eqiad1.wikimedia.cloud. | PRIMARY | 1727442157 | ACTIVE | NONE   |
| 9e14ccb4-0840-4060-b468-2654b8c9e1a8 | svc.eqiad1.wikimedia.cloud.            | PRIMARY | 1732050012 | ACTIVE | NONE   |
| eefa2d49-be26-4ab2-bcbe-b1f0a8fef42c | cloudinfra.eqiad1.wmcloud.org.         | PRIMARY | 1727442153 | ACTIVE | NONE   |
| 4d5aba7a-fa31-4bcf-b863-41bb4c0362c6 | 0-25.56.15.185.in-addr.arpa.           | PRIMARY | 1732882815 | ACTIVE | NONE   |
+--------------------------------------+----------------------------------------+---------+------------+--------+--------+
$ sudo wmcs-openstack --os-project-id=wmflabsdotorg zone list --max-width=80
+--------------------------------------+----------------------------+---------+------------+--------+--------+
| id                                   | name                       | type    |     serial | status | action |
+--------------------------------------+----------------------------+---------+------------+--------+--------+
| 553ef162-add7-4a5c-b115-9cabca662746 | wmflabs.org.               | PRIMARY | 1730307625 | ACTIVE | NONE   |
| ef825cf2-db2d-4480-ad33-2c19b0a188dc | wmflabsdotorg.wmflabs.org. | PRIMARY | 1727442153 | ACTIVE | NONE   |
| e07defe3-6d08-4f37-bd7d-1bdc1c45d7e8 | 56.15.185.in-addr.arpa.    | PRIMARY | 1727442153 | ACTIVE | NONE   |
+--------------------------------------+----------------------------+---------+------------+--------+--------+

Some legacy internal zones are owned by the special noauth-project tenant which novaobserver cannot access. 

$ sudo wmcs-openstack zone list --sudo-project-id noauth-project
+--------------------------------------+---------------------------------+---------+------------+--------+--------+
| id                                   | name                            | type    |     serial | status | action |
+--------------------------------------+---------------------------------+---------+------------+--------+--------+
| 114f1333-c2c1-44d3-beb4-ebed1a91742b | eqiad.wmflabs.                  | PRIMARY | 1732373906 | ACTIVE | NONE   |
| d19aff2d-2d57-4d25-9e26-dab2b3a58be4 | svc.eqiad.wmflabs.              | PRIMARY | 1724360527 | ACTIVE | NONE   |
| e81ea3bb-f8f0-49f4-906b-2d4c2e83cc7e | web.db.svc.eqiad.wmflabs.       | PRIMARY | 1730311096 | ACTIVE | NONE   |
| 04c45c1f-214d-450b-a733-028dcdc87a12 | analytics.db.svc.eqiad.wmflabs. | PRIMARY | 1730311017 | ACTIVE | NONE   |
| 6990e139-49e6-466c-9421-46cf45f05842 | 16.172.in-addr.arpa.            | PRIMARY | 1732885287 | ACTIVE | NONE   |
+--------------------------------------+---------------------------------+---------+------------+--------+--------+


When floating IPs are allocated and assigned, DNS records can be created for them using Horizon: 

$ sudo wmcs-openstack --os-project-id=wmflabsdotorg recordset show wmflabs.org. ntp-01.wmflabs.org.
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| action      | NONE                                 |
| created_at  | 2018-11-02T20:34:18.000000           |
| description | time server for cloud instances      |
| id          | 5812d178-b7a8-4168-bce7-5e064b411f82 |
| name        | ntp-01.wmflabs.org.                  |
| project_id  | wmflabsdotorg                        |
| records     | 185.15.56.3                          |
| status      | ACTIVE                               |
| ttl         | 3600                                 |
| type        | A                                    |
| updated_at  | None                                 |
| version     | 1                                    |
| zone_id     | 553ef162-add7-4a5c-b115-9cabca662746 |
| zone_name   | wmflabs.org.                         |
+-------------+--------------------------------------+

The dns-floating-ip-updater script runs as a systemd timer to ensure that A and PTR records are created for each floating IP. The automatic A record will be of the form instance-$instance.$project.wmflabs.org to make it obvious which IP is served by which instance. 

$ sudo wmcs-openstack --os-project-id=wmflabsdotorg recordset show 56.15.185.in-addr.arpa. 3.56.15.185.in-addr.arpa. --max-width=80
+-------------+----------------------------------------------------------------+
| Field       | Value                                                          |
+-------------+----------------------------------------------------------------+
| action      | NONE                                                           |
| created_at  | 2018-12-17T19:12:11.000000                                     |
| description | MANAGED BY dns-floating-ip-updater.py IN PUPPET - DO NOT       |
|             | UPDATE OR DELETE                                               |
| id          | 5d331e2b-53d1-407e-bf65-ad022ac8e186                           |
| name        | 3.56.15.185.in-addr.arpa.                                      |
| project_id  | wmflabsdotorg                                                  |
| records     | ntp-01.cloudinfra.wmflabs.org.                                 |
|             | ntp-01.wmflabs.org.                                            |
|             | instance-ntp-01.cloudinfra.wmflabs.org.                        |
| status      | ACTIVE                                                         |
| ttl         | None                                                           |
| type        | PTR                                                            |
| updated_at  | 2019-12-12T07:14:47.000000                                     |
| version     | 4                                                              |
| zone_id     | e07defe3-6d08-4f37-bd7d-1bdc1c45d7e8                           |
| zone_name   | 56.15.185.in-addr.arpa.                                        |
+-------------+----------------------------------------------------------------+

The dns-floating-ip-updater script runs as a systemd timer to ensure that lookups for hostnames tied to floating IPs return the associated instance IP when queried via the internal recursors (Split-horizon DNS). 

$ dig +short @ns1.openstack.eqiad1.wikimediacloud.org. irc.beta.wmflabs.org
185.15.56.34
$ dig +short @ns-recursor1.openstack.eqiad1.wikimediacloud.org. irc.beta.wmflabs.org
172.16.4.22

Designate services

Designate has a lot of moving parts. On a given designate hosts, you will (or may) see the following things running:

  • designate-api: the REST endpoint for creating or deleting zones or records. Horizon and the commandline client talk directly to this.
  • designate-sink: listens (via rabbitmq) for nova notifications about instance creation or deletion; creates and deletes records under .eqiad.wmflabs as needed.
  • designate-mdns: a tiny dns server maintained by designate -- this exists mainly as a source of authority for axfr sync requests with powerdns. xfr can only sync records and not create domains.
  • designate-worker: since axfr is only useful for syncing records, designate-pool-manager holds API credentials to update dns zones in powerdns using the pdns API.
  • designate-producer: initiates sync operations performed by designate-worker
  • designate-central: coordinates all of the above, stores stat in the database
  • designate-agent: a leftover service from an older version of designate that doesn't do anything but is somehow still installed by the .deb packges. If it's running, feel free to stop it; puppet doesn't ensure that it's either up or down.
  • memcached/mcrouter: Designate uses tooz backed with memcached to coordinate action between hosts.

The above list is for Designate Bobcat. Services may be rearranged in future releases.

Initial designate/pdns node setup

A lot of the node config is puppetized, but there are several steps that need manual intervention. This is kind of a mess, and more of it should probably be moved into puppet at some point.

Initial puppet runs will complain about the inability to start mariadb. This is complicated on Stretch by the mariadb package being a bit broken. To resolve: 

 root@cloudservices1004:/opt# cd /opt/wmf-mariadb106/
 root@cloudservices1004:/opt/wmf-mariadb10# ./scripts/mariadb-install-db
   Installing MariaDB system tables in '/srv/sqldata' ...

Once puppet runs are clean, create the initial pdns database: 

 root@cloudservices1004:/tmp# mariadb
 MariaDB [(none)]> create database pdns;
 Query OK, 1 row affected (0.00 sec)

Then set up grants: (you'll find the password and the node_ip in /etc/powerdns/pdns.conf, look for gmysql-password and gmysql-host) 

 MariaDB [(none)]> USE pdns;
 Database changed
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'localhost' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.00 sec)
 MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'<node_ip>' IDENTIFIED BY '<password>';
 Query OK, 0 rows affected (0.01 sec)

The pdns-recursor will expect some ip alias files to exist. Those files are maintained by a timer, but to get things running right now, run 

#/usr/local/bin/labs-ip-alias-dump.py

Then restart pdns-recursor.

If adding a new node (as opposed to rebuilding an existing one with a previously registered IP), you'll need to update the designate pool to include the new node. First, dump the current config to /etc/designate/pools.yaml with 

 designate-manage pool generate_file

Edit that file to add the new node, and then: 

 designate-manage pool update  --file /path/to/new/pools.yaml --dry-run

and if everything parses properly, 

   designate-manage pool update  --file /path/to/new/pools.yaml

See also the upstream documentation about the pools.yaml file.

You will also need to update pdns on all nodes to set the 'master' field. This is terrible because of our asymmetric network setup -- when traffic flows from designate to pdns locally it uses a public IP (e.g. 185.15.56.163) but remotely over a private IP (e.g. 172.20.2.4). So each pdns node will have different master records.

As of 2023-11-07, the eqiad records are set like this (these is a SQL commands in the pdns database): 

 update domains set master="172.20.1.5:5354, 185.15.56.162:5354"; # on cloudservices1005.eqiad.wmnet
 update domains set master="172.20.2.4:5354, 185.15.56.163:5354"; # on cloudservices1006.eqiad.wmnet
 update domains set master="<private ipv4 of other pdns node>:5354 <public ipv4 of this pdns node>:5354"; # the general pattern

Note that that means each node will have a different set of masters and this needs updating after domains are migrated between nodes.

Finally, whether this is a newly added node or a rebuild, we need designate to set up the pdns database schema. Instructions about how to set up the pdns schema can be found here: https://doc.powerdns.com/md/authoritative/howtos/#basic-setup-configuring-database-connectivity; here's a copy with a few tweaks (changing large fields from VARCHAR to TEXT to avoid size limits).

Bookworm (pdns 4.7): 

CREATE TABLE domains (
 id                    INT AUTO_INCREMENT,
 name                  VARCHAR(255) NOT NULL,
 master                VARCHAR(128) DEFAULT NULL,
 last_check            INT DEFAULT NULL,
 type                  VARCHAR(8) NOT NULL,
 notified_serial       INT UNSIGNED DEFAULT NULL,
 account               VARCHAR(40) CHARACTER SET 'utf8' DEFAULT NULL,
 options               VARCHAR(64000) DEFAULT NULL,
 catalog               VARCHAR(255) DEFAULT NULL,
 PRIMARY KEY (id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE UNIQUE INDEX name_index ON domains(name);
CREATE INDEX catalog_idx ON domains(catalog);


CREATE TABLE records (
 id                    BIGINT AUTO_INCREMENT,
 domain_id             INT DEFAULT NULL,
 name                  VARCHAR(255) DEFAULT NULL,
 type                  VARCHAR(10) DEFAULT NULL,
 content               TEXT(64000) DEFAULT NULL,
 ttl                   INT DEFAULT NULL,
 prio                  INT DEFAULT NULL,
 disabled              TINYINT(1) DEFAULT 0,
 ordername             VARCHAR(255) BINARY DEFAULT NULL,
 auth                  TINYINT(1) DEFAULT 1,
 PRIMARY KEY (id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE INDEX nametype_index ON records(name,type);
CREATE INDEX domain_id ON records(domain_id);
CREATE INDEX ordername ON records (ordername);


CREATE TABLE supermasters (
 ip                    VARCHAR(64) NOT NULL,
 nameserver            VARCHAR(255) NOT NULL,
 account               VARCHAR(40) CHARACTER SET 'utf8' NOT NULL,
 PRIMARY KEY (ip, nameserver)
) Engine=InnoDB CHARACTER SET 'latin1';


CREATE TABLE comments (
 id                    INT AUTO_INCREMENT,
 domain_id             INT NOT NULL,
 name                  VARCHAR(255) NOT NULL,
 type                  VARCHAR(10) NOT NULL,
 modified_at           INT NOT NULL,
 account               VARCHAR(40) CHARACTER SET 'utf8' DEFAULT NULL,
 comment               TEXT CHARACTER SET 'utf8' NOT NULL,
 PRIMARY KEY (id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE INDEX comments_name_type_idx ON comments (name, type);
CREATE INDEX comments_order_idx ON comments (domain_id, modified_at);


CREATE TABLE domainmetadata (
 id                    INT AUTO_INCREMENT,
 domain_id             INT NOT NULL,
 kind                  VARCHAR(32),
 content               TEXT,
 PRIMARY KEY (id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE INDEX domainmetadata_idx ON domainmetadata (domain_id, kind);


CREATE TABLE cryptokeys (
 id                    INT AUTO_INCREMENT,
 domain_id             INT NOT NULL,
 flags                 INT NOT NULL,
 active                BOOL,
 published             BOOL DEFAULT 1,
 content               TEXT,
 PRIMARY KEY(id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE INDEX domainidindex ON cryptokeys(domain_id);


CREATE TABLE tsigkeys (
 id                    INT AUTO_INCREMENT,
 name                  VARCHAR(255),
 algorithm             VARCHAR(50),
 secret                VARCHAR(255),
 PRIMARY KEY (id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE UNIQUE INDEX namealgoindex ON tsigkeys(name, algorithm);

Bullseye: 

CREATE TABLE domains (
 id                    INT AUTO_INCREMENT,
 name                  VARCHAR(255) NOT NULL,
 master                VARCHAR(128) DEFAULT NULL,
 last_check            INT DEFAULT NULL,
 type                  VARCHAR(6) NOT NULL,
 notified_serial       INT DEFAULT NULL,
 account               VARCHAR(40) DEFAULT NULL,
 PRIMARY KEY (id)
) Engine=InnoDB;
CREATE UNIQUE INDEX name_index ON domains(name);
CREATE TABLE records (
 id                    BIGINT AUTO_INCREMENT,
 domain_id             INT DEFAULT NULL,
 name                  VARCHAR(255) DEFAULT NULL,
 type                  VARCHAR(10) DEFAULT NULL,
 content               TEXT(64000) DEFAULT NULL,
 ttl                   INT DEFAULT NULL,
 prio                  INT DEFAULT NULL,
 change_date           INT DEFAULT NULL,
 disabled              TINYINT(1) DEFAULT 0,
 ordername             VARCHAR(255) BINARY DEFAULT NULL,
 auth                  TINYINT(1) DEFAULT 1,
 PRIMARY KEY (id)
) Engine=InnoDB;
CREATE INDEX nametype_index ON records(name,type);
CREATE INDEX domain_id ON records(domain_id);
CREATE INDEX recordorder ON records (domain_id, ordername);
CREATE TABLE supermasters (
 ip                    VARCHAR(64) NOT NULL,
 nameserver            VARCHAR(255) NOT NULL,
 account               VARCHAR(40) NOT NULL,
 PRIMARY KEY (ip, nameserver)
) Engine=InnoDB;
CREATE TABLE comments (
 id                    INT AUTO_INCREMENT,
 domain_id             INT NOT NULL,
 name                  VARCHAR(255) NOT NULL,
 type                  VARCHAR(10) NOT NULL,
 modified_at           INT NOT NULL,
 account               VARCHAR(40) NOT NULL,
 comment               TEXT(21844) NOT NULL,
 PRIMARY KEY (id)
) Engine=InnoDB;
CREATE INDEX comments_domain_id_idx ON comments (domain_id);
CREATE INDEX comments_name_type_idx ON comments (name, type);
CREATE INDEX comments_order_idx ON comments (domain_id, modified_at);
CREATE TABLE domainmetadata (
 id                    INT AUTO_INCREMENT,
 domain_id             INT NOT NULL,
 kind                  VARCHAR(32),
 content               TEXT,
 PRIMARY KEY (id)
) Engine=InnoDB;
CREATE INDEX domainmetadata_idx ON domainmetadata (domain_id, kind);
CREATE TABLE cryptokeys (
 id                    INT AUTO_INCREMENT,
 domain_id             INT NOT NULL,
 flags                 INT NOT NULL,
 active                BOOL,
 content               TEXT,
 PRIMARY KEY(id)
) Engine=InnoDB;
CREATE INDEX domainidindex ON cryptokeys(domain_id);
CREATE TABLE tsigkeys (
 id                    INT AUTO_INCREMENT,
 name                  VARCHAR(255),
 algorithm             VARCHAR(50),
 secret                VARCHAR(255),
 PRIMARY KEY (id)
) Engine=InnoDB;
CREATE UNIQUE INDEX namealgoindex ON tsigkeys(name, algorithm);


Now we're ready to restart all designate services (and pdns) and see what's broken. Rebooting is easiest!

Now pdns will probably be complaining (in the syslog) about getting axfr requests for unknown domains. That's because designate doesn't automatically populate domains in pdns, it only creates them as needed when records are added. We have too many domains to wait, so the best way forward is to dump the 'domains' table from an existing node and import it on the new node. 

 #  # on existing, working node:
 # mariadb-dump pdns domains > domains.sql
 #  # on new node
 # mariadb pdns < domains.sql

Finally: designate-sink needs to be able to talk to the nova proxy, in order to clean up proxies associated with a deleted VM. Add access to port 5668 for the IP of the new node to project-proxy's security group. Similarly, when creating a new node make sure it can talk to the cloud puppetmasters on port 8101 so it can clean up the certs associated with a deleted VM.

Additional consideration: The galera DB (cloudcontrols) need a DB GRANT for haproxy nodes (cloudlb) for them to access designate tables. This is because some designate services (maybe designate-central in cloudservices boxes) want to talk to the galera DB and they do via haproxy (cloudlb).

This GRANT takes this form: 

# for cloudlb1001.private.eqiad.wikimedia.cloud
GRANT ALL PRIVILEGES ON designate.* TO 'designate'@'172.20.1.2'  IDENTIFIED BY '<redacted>';
# for cloudlb1002.private.eqiad.wikimedia.cloud
GRANT ALL PRIVILEGES ON designate.* TO 'designate'@'172.20.2.2'  IDENTIFIED BY '<redacted>';

If all is well there should be a ready-made script on cloudcontrols to set up these grants: /etc/designate/designate_grants.mysql

[..] </syntaxhighlight>

Basic operations

New PTR record

Example command to create a new PTR record: 

# openstack --os-project-id cloudinfra-codfw1dev recordset create --type PTR --record test.codfw1dev.wmcloud.org. --description "test" 0-29.57.15.185.in-addr.arpa. 1.0-29.57.15.185.in-addr.arpa.
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| action      | CREATE                               |
| created_at  | 2020-05-25T17:07:34.000000           |
| description | test                                 |
| id          | 450a83c9-cca2-4c82-8a28-13cf035c7702 |
| name        | 1.0-29.57.15.185.in-addr.arpa.       |
| project_id  | cloudinfra-codfw1dev                 |
| records     | test.codfw1dev.wmcloud.org.          |
| status      | PENDING                              |
| ttl         | None                                 |
| type        | PTR                                  |
| updated_at  | None                                 |
| version     | 1                                    |
| zone_id     | 7796b60e-7bc6-4cb4-9e4d-dae039d3f912 |
| zone_name   | 0-29.57.15.185.in-addr.arpa.         |
+-------------+--------------------------------------+

In previous openstack versions we were using the designate CLI, but as of Rocky this probably no longer works: 

root@cloudcontrol1003:~# designate --all-tenants --os-project-name admin record-create 16.172.in-addr.arpa. \
>  --name 1.0.16.172.in-addr.arpa. --type PTR --data cloudinstances2b-gw.svc.eqiad.wmflabs. \
>  --description "Neutron virtual router. Record created by hand"
+-------------+------------------------------------------------+
| Field       | Value                                          |
+-------------+------------------------------------------------+
| description | Neutron virtual router. Record created by hand |
| type        | PTR                                            |
| created_at  | 2018-11-30T13:26:47.000000                     |
| updated_at  | None                                           |
| domain_id   | 6990e139-49e6-466c-9421-46cf45f05842           |
| priority    | None                                           |
| ttl         | None                                           |
| data        | cloudinstances2b-gw.svc.eqiad.wmflabs.       |
| id          | 855a3d4d-4caf-4652-b897-d6559a64bb4e           |
| name        | 1.0.16.172.in-addr.arpa.                       |
+-------------+------------------------------------------------+

New A records

Example command to create a new A record: 

root@cloudcontrol1003:~# OS_PROJECT_ID=tools openstack recordset create --type A --records 172.16.0.24 --sudo-project-id tools svc.tools.eqiad1.wikimedia.cloud. elasticsearch
+-------------+-------------------------------------------------+
| Field       | Value                                           |   
+-------------+-------------------------------------------------+
| action      | CREATE                                          |   
| created_at  | 2020-03-03T18:15:41.000000                      |   
| description | None                                            |   
| id          | 35d2a87c-d355-4805-90d9-4a088f5981bc            |   
| name        | elasticsearch.svc.tools.eqiad1.wikimedia.cloud. |
| project_id  | tools                                           |   
| records     | 172.16.0.24                                     |   
| status      | PENDING                                         |   
| ttl         | None                                            |   
| type        | A                                               |   
| updated_at  | None                                            |   
| version     | 1                                               |   
| zone_id     | 808263bd-522d-42c3-88a1-40af24b2929e            |   
| zone_name   | svc.tools.eqiad1.wikimedia.cloud.               |   
+-------------+-------------------------------------------------+

New NS record

Example command for creating a new NS record by hand (sub domain delegation): 

root@cloudcontrol1004:~# designate --all-tenants --os-project-name cloudinfra record-create wmcloud.org. --name codfw1dev.wmcloud.org --type NS --data ns0.openstack.codfw1dev.wikimediacloud.org --description "delegation to Designate in the other deployment. Record created by hand"
+-------------+-------------------------------------------------------------------------+
| Field       | Value                                                                   |
+-------------+-------------------------------------------------------------------------+
| description | delegation to Designate in the other deployment. Record created by hand |
| type        | NS                                                                      |
| created_at  | 2020-01-28T10:02:28.000000                                              |
| updated_at  | None                                                                    |
| domain_id   | a5f22422-815c-43d2-95aa-8fbdee95de01                                    |
| priority    | None                                                                    |
| ttl         | None                                                                    |
| data        | ns0.openstack.codfw1dev.wikimediacloud.org.                             |
| id          | a0a86d45-6bab-445f-8bd2-f0bf2f76e45a                                    |
| name        | codfw1dev.wmcloud.org.                                                  |
+-------------+-------------------------------------------------------------------------+

Delegations within the same openstack deployment are usually made using the wmcs-makedomain script.

Updating an existing record

Example command for updating an existing A record: 

# Update A record:
# designate --all-tenants --os-project-name admin record-update --name tools.db.svc.eqiad.wmflabs. --type A --data 185.15.56.54 --description "ToolsDB server. Record created by hand" df88fcb3-fbc2-42f1-bb12-2424c8b7117e 522952da-b43b-4032-bcb5-df04b6a1cdbc
# which means (uids translated):
# designate --all-tenants --os-project-name admin record-update --name tools.db.svc.eqiad.wmflabs. --type A --data 185.15.56.54 --description "ToolsDB server. Record created by hand" db.svc.eqiad.wmflabs. tools.db.svc.eqiad.wmflabs.

TODO: verify me and use proper syntax highlighting.

Maintenance operations

If things are misbehaving

Restart everything

The bits of designate do their best to remain synchronized and organized, but it's complicated with some races and leaks. If things are misbehaving, the first step is always to just restart everything.

We have a cookbook for that! 

sudo cookbook wmcs.openstack.restart_openstack --designate --cluster-name eqiad1

Check the designate-sink logs

If new VMs aren't getting their A records (usually first noticed via a failure in the fullstack tests) and restarting everything didn't help, look in /var/log/designate/designate-sink.log to see what's happening. Normal operations like creation and deletion will show up there as warnings, just so we can tell what's going on.

Look at pdns

If all else fails and new zones aren't handled properly, there may be an issue with AXFR or the PDNS api. PDNS logs to syslog on the cloudservices nodes; you can see its logs with 

tail -f /var/log/syslog | grep -i pdns

Cleaning up leaked records

In some cases, leaked DNS records may happen. See nova-fullstack for information on the VM testing that is done every 5 minutes. If you see a lot of fullstackd* DNS entries on the cloudinfra project in horizon, DNS records have leaked.

We have a custom script to detect/correct them: wmcs-dnsleaks. If this script is run with the --delete argument, it will delete leaked records, which is useful if there are many of them. 

andrew@cloudcontrol2001-dev:~$ sudo wmcs-dnsleaks
checking zone: 16.172.in-addr.arpa.
Found 2 ptr recordsets for the same VM: canary2002-dev-2.cloudvirt-canary.codfw1dev.wikimedia.cloud. ['38.128.16.172.in-addr.arpa.', '170.128.16.172.in-addr.arpa.']
Found 2 ptr recordsets for the same VM: canary2003-dev-2.cloudvirt-canary.codfw1dev.wikimedia.cloud. ['134.128.16.172.in-addr.arpa.', '131.128.16.172.in-addr.arpa.']
checking zone: codfw1dev.wikimedia.cloud.
skipping public zone: codfw1dev.wmcloud.org.
skipping public zone: cloudinfra-codfw1dev.codfw1dev.wmcloud.org.
checking zone: 57.15.185.in-addr.arpa.
checking zone: 0-29.57.15.185.in-addr.arpa.

Manual fix for multiple IP records

wmcs-dnsleaks will print a message like this when it finds an A record with multiple IPs: 

A record for canary1038-01.cloudvirt-canary.eqiad.wmflabs. has multiple IPs: ['172.16.2.37', '172.16.2.54']
This needs cleanup but that isn't implemented and almost never happens.

These can be manually fixed: 

$ ssh cloudcontrol1003.wikimedia.org
$ sudo su -
$ source /root/novaenv.sh
$ openstack recordset set eqiad.wmflabs. canary1038-01.cloudvirt-canary.eqiad.wmflabs. --record 172.16.2.54 --edit-managed --sudo-project-id noauth-project
url: /zones/114f1333-c2c1-44d3-beb4-ebed1a91742b/recordsets/0464e35f-92a0-46f5-a31e-b5dde9fb5ea7
kwargs: {'data': '{"records": ["172.16.2.54"]}'}
+-------------+-----------------------------------------------+
| Field       | Value                                         |
+-------------+-----------------------------------------------+
| action      | UPDATE                                        |
| created_at  | 2020-07-31T22:43:37.000000                    |
| description | None                                          |
| id          | 0464e35f-92a0-46f5-a31e-b5dde9fb5ea7          |
| name        | canary1038-01.cloudvirt-canary.eqiad.wmflabs. |
| project_id  | noauth-project                                |
| records     | 172.16.2.54                                   |
| status      | PENDING                                       |
| ttl         | None                                          |
| type        | A                                             |
| updated_at  | 2020-08-28T21:05:51.000000                    |
| version     | 2                                             |
| zone_id     | 114f1333-c2c1-44d3-beb4-ebed1a91742b          |
| zone_name   | eqiad.wmflabs.                                |
+-------------+-----------------------------------------------+

DNS zone creating / transfers

If the zone you want to create doesn't have a parent zone already registered in Designate (i.e, is not a sub-zone), you can create the zone directly in the project it will live on: 

# openstack --os-project-id cloudinfra-codfw1dev zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description "floating IPs subnet" 0-29.57.15.185.in-addr.arpa.
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| action         | CREATE                               |
| attributes     |                                      |
| created_at     | 2020-05-25T16:34:38.000000           |
| description    | floating IPs subnet                  |
| email          | root@wmflabs.org                     |
| id             | 7796b60e-7bc6-4cb4-9e4d-dae039d3f912 |
| masters        |                                      |
| name           | 0-29.57.15.185.in-addr.arpa.         |
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id     | cloudinfra-codfw1dev                 |
| serial         | 1590424478                           |
| status         | PENDING                              |
| transferred_at | None                                 |
| ttl            | 3600                                 |
| type           | PRIMARY                              |
| updated_at     | None                                 |
| version        | 1                                    |
+----------------+--------------------------------------+

But anyway, according to our DNS setup, certain domains (zones in Designate) should be created in the admin project and then transfered / relocated to the cloudinfra project.

This can be done with the openstack native transfer mechanism. 

# openstack zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description "floating IPs subnet" 57.15.185.in-addr.arpa.
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| action         | CREATE                               |
| attributes     |                                      |
| created_at     | 2020-04-06T17:38:44.000000           |
| description    | floating IPs subnet                  |
| email          | root@wmflabs.org                     |
| id             | 24e82029-c970-4422-875f-e81d9e59e33a |
| masters        |                                      |
| name           | 57.15.185.in-addr.arpa.              |
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id     | admin                                |
| serial         | 1586194724                           |
| status         | PENDING                              |
| transferred_at | None                                 |
| ttl            | 3600                                 |
| type           | PRIMARY                              |
| updated_at     | None                                 |
| version        | 1                                    |
+----------------+--------------------------------------+

# openstack zone transfer request create --target-project-id cloudinfra 67603ef4-3d64-40d6-90d3-5b7776a99034
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                               |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| created_at        | 2020-02-18T15:35:43.000000                                                                                                          |
| description       | None                                                                                                                                |
| id                | 0a30d1fe-51ea-4da4-8a27-80c16e681ff6                                                                                                |
| key               | LOYRJY7Q                                                                                                                            |
| links             | {u'self': u'http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones/tasks/transfer_requests/0a30d1fe-51ea-4da4-8a27-80c16e681ff6'} |
| project_id        | admin                                                                                                                               |
| status            | ACTIVE                                                                                                                              |
| target_project_id | cloudinfra                                                                                                                          |
| updated_at        | None                                                                                                                                |
| zone_id           | 67603ef4-3d64-40d6-90d3-5b7776a99034                                                                                                |
| zone_name         | eqiad1.wikimedia.cloud.                                                                                                             |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+

# OS_PROJECT_ID=cloudinfra openstack zone transfer accept request --key LOYRJY7Q --transfer-id 0a30d1fe-51ea-4da4-8a27-80c16e681ff6 
[..]

# OS_PROJECT_ID=cloudinfra openstack zone show 67603ef4-3d64-40d6-90d3-5b7776a99034
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| action         | NONE                                 |
| attributes     | {}                                   |
| created_at     | 2019-10-18T15:57:51.000000           |
| description    | None                                 |
| email          | root@wmflabs.org                     |
| id             | 67603ef4-3d64-40d6-90d3-5b7776a99034 |
| masters        |                                      |
| name           | eqiad1.wikimedia.cloud.              |
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id     | cloudinfra                           |
| serial         | 1581676303                           |
| status         | ACTIVE                               |
| transferred_at | None                                 |
| ttl            | 3600                                 |
| type           | PRIMARY                              |
| updated_at     | 2020-02-18T15:38:17.000000           |
| version        | 6                                    |
+----------------+--------------------------------------+

See phabricator T245494 - CloudVPS: figure out DNS zone ownership transfers and setup for context on how this was figured out.


Zones

tools.eqiad.wmflabs

Some special hosts are defined in profile::openstack::main::pdns::recursor_aliaser_extra_records:

  • tools-db.tools.eqiad.wmflabs
  • tools-redis.tools.eqiad.wmflabs
  • tools-redis.eqiad.wmflabs
  • puppet

.wmflabs and .wikimedia.cloud

Records created by designate-sink are marked as 'managed' which means they can not be edited with Horizon or normal command line actions. This can be overridden with the --edit-managed flag, for example: 

# OS_PROJECT_ID=cloudinfra openstack recordset set eqiad1.wikimedia.cloud. tools-redis-1003.tools.eqiad1.wikimedia.cloud. --record 172.16.1.107 --edit-managed

Records under .wmflabs (and a few other things including the ptr zone for internal IPs, 16.172.in-addr.arpa) are in a special visible-only-to-Designate tenant, 'noauth-project'. Horizon and many command-line tools are unable to access that tenant, but you can see them with the 'novaadmin' user and the --sudo-project-id flag: 

# openstack zone list --sudo-project-id noauth-project

Records under .wikimedia.cloud are owned by the 'cloudinfra' project and can be viewed there. They are also managed, though, so modifying them requires use of the commandline and --edit-managed, as above.

svc zone in .wikimedia.cloud domain

To create and delegate the svc.NEWPROJECT.eqiad1.wikimedia.cloud domain, do the following. Please note that the parent domain always belong to the cloudinfra project per our setup.

  • Create the DNS zone for newproject in the cloudinfra project:
user@cloudcontrol1004:~ $ sudo wmcs-openstack --os-project-id cloudinfra zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 svc.newproject.eqiad1.wikimedia.cloud.
+----------------+---------------------------------------+
| Field          | Value                                 |
+----------------+---------------------------------------+
| action         | CREATE                                |
| attributes     |                                       |
| created_at     | 2020-05-21T12:27:05.000000            |
| description    | None                                  |
| email          | root@wmflabs.org                      |
| id             | 345c1fbc-4f15-4ab5-b87e-38bd284b55b9  |
| masters        |                                       |
| name           | svc.newproject.eqiad1.wikimedia.cloud.|
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842  |
| project_id     | cloudinfra                            |
| serial         | 1590064025                            |
| status         | PENDING                               |
| transferred_at | None                                  |
| ttl            | 3600                                  |
| type           | PRIMARY                               |
| updated_at     | None                                  |
| version        | 1                                     |
+----------------+---------------------------------------+
  • create the zone transfer request, mind the project specifications:
user@cloudcontrol1004:~ $ sudo wmcs-openstack --os-project-id cloudinfra zone transfer request create --target-project-id newproject 345c1fbc-4f15-4ab5-b87e-38bd284b55b9
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                             |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| created_at        | 2020-05-21T12:28:39.000000                                                                                                        |
| description       | None                                                                                                                              |
| id                | 3910d76b-b660-42a4-b995-21db53e3e42b                                                                                              |
| key               | KQBAHXJ4                                                                                                                          |
| links             | {'self': 'http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones/tasks/transfer_requests/3910d76b-b660-42a4-b995-21db53e3e42b'} |
| project_id        | cloudinfra                                                                                                                        |
| status            | ACTIVE                                                                                                                            |
| target_project_id | paws                                                                                                                              |
| updated_at        | None                                                                                                                              |
| zone_id           | 345c1fbc-4f15-4ab5-b87e-38bd284b55b9                                                                                              |
| zone_name         | svc.newproject.eqiad1.wikimedia.cloud.                                                                                            |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------+
  • accept the zone transfer request, again mind the project specifications:
user@cloudcontrol1004:~ $ sudo wmcs-openstack --os-project-id newproject zone transfer accept request --key G3VR0YQR --transfer-id 5d3367a7-2d3c-4ed7-8e44-8107ee9d7520
[...]
  • done! confirm the new ownership:
user@cloudcontrol1004:~ $ sudo wmcs-openstack zone show 345c1fbc-4f15-4ab5-b87e-38bd284b55b9 --os-project-id paws
+----------------+----------------------------------------+
| Field          | Value                                  |
+----------------+----------------------------------------+
| action         | NONE                                   |
| attributes     |                                        |
| created_at     | 2020-05-21T12:27:05.000000             |
| description    | None                                   |
| email          | root@wmflabs.org                       |
| id             | 345c1fbc-4f15-4ab5-b87e-38bd284b55b9   |
| masters        |                                        |
| name           | svc.newproject.eqiad1.wikimedia.cloud. |
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842   |
| project_id     | newproject                             |
| serial         | 1590064025                             |
| status         | ACTIVE                                 |
| transferred_at | None                                   |
| ttl            | 3600                                   |
| type           | PRIMARY                                |
| updated_at     | 2020-05-21T12:32:45.000000             |
| version        | 4                                      |
+----------------+----------------------------------------+

The instructions here are specific to this use case, including details on the ownership dancing, rather than the generic instructions in the DNS zone creating / transfers section.

Administrative scripts

See CloudVPS maintenance, DNS admin scripts.

Wiki Replica DNS

See Portal:Data_Services/Admin/Wiki_Replica_DNS

labs-ip-alias-dump

This script actually generates a lua script which is read by pdns-recursor. This is not technical anything related to Designate, but it is relevant to Cloud VPS DNS lookups. The script hooks two lua resolver hooks:

postresolve
Maps public IPs to private IPs so that a lookup that results in a floating IP instead returns the private IP of the instance that it is attached to.
preresolve
Injects additional DNS results that are only visible via the recursor. Currently used to provide DNS responses for:
  • puppet.
  • tools-db.tools.eqiad.wmflabs.
  • tools-redis.eqiad.wmflabs.
  • tools-redis.tools.eqiad.wmflabs.

The script runs via cron every 60 minutes on each pdns-recursor host.

dns-floating-ip-updater

We have a script called dns-floating-ip-updater that sets up instance-{instancename}.{project}.wmcloud.org records and the reverse DNS record for that, then also sets up the reverse DNS for all manually curated A records.

OpenStack public (v4) names use 185.15.56.0/25) (with the latter /25 used for infrastructure with DNS managed via Netbox), so the we use RFC 2317-style delegation: We have a zone called 0-25.56.15.185.in-addr.arpa. (living under the cloudinfra tenant) that production CNAMEs all the usual in-addr records into, and delegates the zone to Cloud VPS nameservers.

What happens when someone makes a reverse DNS lookup, for let's say 185.15.56.12:

  • It gets turned into a PTR lookup on 12.56.15.185.in-addr.arpa. as normal
  • Prod NS servers control the 56.15.185.in-addr.arpa. zone - if you open the file for that in the operations/dns.git repository ('templates' directory), you'll see normal Netbox includes and such
  • So under 185.15.56, number 12 is IN CNAME 12.0-25 i.e. 12.0-25.56.15.185.in-addr.arpa.
  • 0-25.56.15.185.in-addr.arpa. is delegated to Designate.
  • So the full lookup becomes 12.0-25.56.15.185.in-addr.arpa. and is sent to the Cloud VPS auth DNS servers
  • This script ensures that designate knows how to answer that query, by populating all the records.

For example: 

$ host mail.toolsbeta.wmcloud.org
bastion.wmflabs.org has address 208.80.155.129 # this is answered by designate as they control wmcloud.org and its subdomains
$ host 185.15.56.12
12.56.15.185.in-addr.arpa is an alias for 12.0-25.56.15.185.in-addr.arpa. # this is the production nameservers delegating the query to designate
12.0-25.56.15.185.in-addr.arpa domain name pointer mail.toolsbeta.wmcloud.org. # this is from designate and is because mail.toolsbeta.wmcloud.org points to that IP
12.0-25.56.15.185.in-addr.arpa domain name pointer instance-toolsbeta-mail-2.toolsbeta.wmcloud.org. # this comes from designate and is used to indicate which instance a given IP points to, partially in case it lacks other names, but it's helpful regardless. (think what happens if we didn't have this entry but you wanted to go look up the instance on the project's instance list knowing only it's IP - and that's assuming you know which project to look under)
$ host instance-toolsbeta-mail-2.toolsbeta.wmcloud.org
instance-toolsbeta-mail-2.toolsbeta.wmcloud.org has address 185.15.56.12 # this comes from designate - wouldn't make sense to provide the PTR above without this

We will need to do some work on our script to be able to add new domains or public IP ranges (e.g. for other datacentres, to introduce IPv6, or to simply get more addresses).

Announcement post: https://www.mail-archive.com/labs-l@lists.wikimedia.org/msg04516.html

See also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/DNS/Designate&oldid=2249499"