Portal:Cloud VPS/Admin/DNS/Designate
Cloud VPS DNS is PowerDNS, backed by a database controlled by OpenStack Designate. Designate services run on cloudcontrol node and use the same galera cluster database as other openstack services. PowerDNS runs on cloudservices boxes, and each instance has its own local mariadb database.
Basic info
When an instance is created a DNS A record is created in Designate.
$ dig +noall +answer tools-sgebastion-08.tools.eqiad1.wikimedia.cloud
tools-sgebastion-08.tools.eqiad1.wikimedia.cloud. 3541 IN A 172.16.3.190
Root zones for public facing DNS records are owned by either the cloudinfra or wmflabsdotorg tenants:
$ sudo wmcs-openstack --os-project-id=cloudinfra zone list --max-width=80
+------------------+------------------+---------+------------+--------+--------+
| id | name | type | serial | status | action |
+------------------+------------------+---------+------------+--------+--------+
| d863546e-cde7-4a | cloudinfra.wmfla | PRIMARY | 1583906708 | ACTIVE | NONE |
| 4c-b2e2-a7e89db0 | bs.org. | | | | |
| f196 | | | | | |
| 67603ef4-3d64-40 | eqiad1.wikimedia | PRIMARY | 1589574668 | ACTIVE | NONE |
| d6-90d3-5b7776a9 | .cloud. | | | | |
| 9034 | | | | | |
| 6ddcb082-69d6-43 | wmcloud.org. | PRIMARY | 1585587288 | ACTIVE | NONE |
| f4-9993-5c6bdc27 | | | | | |
| dfc9 | | | | | |
+------------------+------------------+---------+------------+--------+--------+
$ sudo wmcs-openstack --os-project-id=wmflabsdotorg zone list --max-width=80
+------------------+------------------+---------+------------+--------+--------+
| id | name | type | serial | status | action |
+------------------+------------------+---------+------------+--------+--------+
| 553ef162-add7-4a | wmflabs.org. | PRIMARY | 1589567523 | ACTIVE | NONE |
| 5c-b115-9cabca66 | | | | | |
| 2746 | | | | | |
| ef825cf2-db2d-44 | wmflabsdotorg.wm | PRIMARY | 1583906762 | ACTIVE | NONE |
| 80-ad33-2c19b0a1 | flabs.org. | | | | |
| 88dc | | | | | |
| 933a78c2-3d8d-4e | 128-25.155.80.20 | PRIMARY | 1583906762 | ACTIVE | NONE |
| e1-bbef-9ab30be5 | 8.in-addr.arpa. | | | | |
| f972 | | | | | |
| e07defe3-6d08-4f | 56.15.185.in- | PRIMARY | 1589491014 | ACTIVE | NONE |
| 37-bd7d-1bdc1c45 | addr.arpa. | | | | |
| d7e8 | | | | | |
+------------------+------------------+---------+------------+--------+--------+
Reverse zones (and some legacy internal zones) are owned by the special noauth-project tenant which novaobserver cannot access.
$ sudo wmcs-openstack zone list --sudo-project-id noauth-project --fit-width
+------------------+------------------+---------+------------+--------+--------+
| id | name | type | serial | status | action |
+------------------+------------------+---------+------------+--------+--------+
| 114f1333-c2c1-44 | eqiad.wmflabs. | PRIMARY | 1589574227 | ACTIVE | NONE |
| d3-beb4-ebed1a91 | | | | | |
| 742b | | | | | |
| 8d114f3c-815b-46 | 68.10.in- | PRIMARY | 1582041663 | ACTIVE | NONE |
| 6c-bdd4-9b91f704 | addr.arpa. | | | | |
| ea60 | | | | | |
| d19aff2d-2d57-4d | svc.eqiad.wmflab | PRIMARY | 1585953047 | ACTIVE | NONE |
| 25-9e26-dab2b3a5 | s. | | | | |
| 8be4 | | | | | |
| df88fcb3-fbc2-42 | db.svc.eqiad.wmf | PRIMARY | 1583876852 | ACTIVE | NONE |
| f1-bb12-2424c8b7 | labs. | | | | |
| 117e | | | | | |
| e81ea3bb-f8f0-49 | web.db.svc.eqiad | PRIMARY | 1586976118 | ACTIVE | NONE |
| f4-906b-2d4c2e83 | .wmflabs. | | | | |
| cc7e | | | | | |
| 04c45c1f-214d-45 | analytics.db.svc | PRIMARY | 1586975792 | ACTIVE | NONE |
| 0b-a733-028dcdc8 | .eqiad.wmflabs. | | | | |
| 7a12 | | | | | |
| 6990e139-49e6-46 | 16.172.in- | PRIMARY | 1589574225 | ACTIVE | NONE |
| 6c-9421-46cf45f0 | addr.arpa. | | | | |
| 5842 | | | | | |
+------------------+------------------+---------+------------+--------+--------+
When floating IPs are allocated and assigned, DNS records can be created for them using Horizon:
$ sudo wmcs-openstack --os-project-id=wmflabsdotorg recordset show wmflabs.org. ntp-01.wmflabs.org.
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| action | NONE |
| created_at | 2018-11-02T20:34:18.000000 |
| description | time server for cloud instances |
| id | 5812d178-b7a8-4168-bce7-5e064b411f82 |
| name | ntp-01.wmflabs.org. |
| project_id | wmflabsdotorg |
| records | 185.15.56.3 |
| status | ACTIVE |
| ttl | 3600 |
| type | A |
| updated_at | None |
| version | 1 |
| zone_id | 553ef162-add7-4a5c-b115-9cabca662746 |
| zone_name | wmflabs.org. |
+-------------+--------------------------------------+
The dns-floating-ip-updater script runs as a systemd timer to ensure that A and PTR records are created for each floating IP. The automatic A record will be of the form instance-$instance.$project.wmflabs.org to make it obvious which IP is served by which instance.
$ sudo wmcs-openstack --os-project-id=wmflabsdotorg recordset show 56.15.185.in-addr.arpa. 3.56.15.185.in-addr.arpa. --max-width=80
+-------------+----------------------------------------------------------------+
| Field | Value |
+-------------+----------------------------------------------------------------+
| action | NONE |
| created_at | 2018-12-17T19:12:11.000000 |
| description | MANAGED BY dns-floating-ip-updater.py IN PUPPET - DO NOT |
| | UPDATE OR DELETE |
| id | 5d331e2b-53d1-407e-bf65-ad022ac8e186 |
| name | 3.56.15.185.in-addr.arpa. |
| project_id | wmflabsdotorg |
| records | ntp-01.cloudinfra.wmflabs.org. |
| | ntp-01.wmflabs.org. |
| | instance-ntp-01.cloudinfra.wmflabs.org. |
| status | ACTIVE |
| ttl | None |
| type | PTR |
| updated_at | 2019-12-12T07:14:47.000000 |
| version | 4 |
| zone_id | e07defe3-6d08-4f37-bd7d-1bdc1c45d7e8 |
| zone_name | 56.15.185.in-addr.arpa. |
+-------------+----------------------------------------------------------------+
The dns-floating-ip-updater script runs as a systemd timer to ensure that lookups for hostnames tied to floating IPs return the associated instance IP when queried via the internal recursors (Split-horizon DNS).
$ dig +short @ns1.openstack.eqiad1.wikimediacloud.org. irc.beta.wmflabs.org
185.15.56.34
$ dig +short @ns-recursor1.openstack.eqiad1.wikimediacloud.org. irc.beta.wmflabs.org
172.16.4.22
Designate services
Designate has a lot of moving parts. On a given designate hosts, you will (or may) see the following things running:
- designate-api: the REST endpoint for creating or deleting zones or records. Horizon and the commandline client talk directly to this.
- designate-sink: listens (via rabbitmq) for nova notifications about instance creation or deletion; creates and deletes records under .eqiad.wmflabs as needed.
- designate-mdns: a tiny dns server maintained by designate -- this exists mainly as a source of authority for axfr sync requests with powerdns. xfr can only sync records and not create domains.
- designate-worker: since axfr is only useful for syncing records, designate-pool-manager holds API credentials to update dns zones in powerdns using the pdns API.
- designate-producer: initiates sync operations performed by designate-worker
- designate-central: coordinates all of the above, stores stat in the database
- designate-agent: a leftover service from an older version of designate that doesn't do anything but is somehow still installed by the .deb packges. If it's running, feel free to stop it; puppet doesn't ensure that it's either up or down.
- memcached/mcrouter: Designate uses tooz backed with memcached to coordinate action between hosts.
The above list is for Designate Bobcat. Services may be rearranged in future releases.
Initial designate/pdns node setup
A lot of the node config is puppetized, but there are several steps that need manual intervention. This is kind of a mess, and more of it should probably be moved into puppet at some point.
Initial puppet runs will complain about the inability to start mariadb. This is complicated on Stretch by the mariadb package being a bit broken. To resolve:
root@cloudservices1004:/opt# cd /opt/wmf-mariadb106/ root@cloudservices1004:/opt/wmf-mariadb10# ./scripts/mariadb-install-db Installing MariaDB system tables in '/srv/sqldata' ...
Once puppet runs are clean, create the initial pdns database:
root@cloudservices1004:/tmp# mariadb MariaDB [(none)]> create database pdns; Query OK, 1 row affected (0.00 sec)
Then set up grants: (you'll find the password and the node_ip in /etc/powerdns/pdns.conf, look for gmysql-password and gmysql-host)
MariaDB [(none)]> USE pdns; Database changed MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'localhost' IDENTIFIED BY '<password>'; Query OK, 0 rows affected (0.00 sec) MariaDB [pdns]> GRANT ALL ON pdns.* TO 'pdns'@'<node_ip>' IDENTIFIED BY '<password>'; Query OK, 0 rows affected (0.01 sec)
The pdns-recursor will expect some ip alias files to exist. Those files are maintained by a timer, but to get things running right now, run
#/usr/local/bin/labs-ip-alias-dump.py
Then restart pdns-recursor.
If adding a new node (as opposed to rebuilding an existing one with a previously registered IP), you'll need to update the designate pool to include the new node. First, dump the current config to /etc/designate/pools.yaml with
designate-manage pool generate_file
Edit that file to add the new node, and then:
designate-manage pool update --file /path/to/new/pools.yaml --dry-run
and if everything parses properly,
designate-manage pool update --file /path/to/new/pools.yaml
See also the upstream documentation about the pools.yaml file.
You will also need to update pdns on all nodes to set the 'master' field. This is terrible because of our asymmetric network setup -- when traffic flows from designate to pdns locally it uses a public IP (e.g. 185.15.56.163) but remotely over a private IP (e.g. 172.20.2.4). So each pdns node will have different master records.
As of 2023-11-07, the eqiad records are set like this (these is a SQL commands in the pdns database):
update domains set master="172.20.1.5:5354, 185.15.56.162:5354"; # on cloudservices1005.eqiad.wmnet update domains set master="172.20.2.4:5354, 185.15.56.163:5354"; # on cloudservices1006.eqiad.wmnet update domains set master="<private ipv4 of other pdns node>:5354 <public ipv4 of this pdns node>:5354"; # the general pattern
Note that that means each node will have a different set of masters and this needs updating after domains are migrated between nodes.
Finally, whether this is a newly added node or a rebuild, we need designate to set up the pdns database schema. Instructions about how to set up the pdns schema can be found here: https://doc.powerdns.com/md/authoritative/howtos/#basic-setup-configuring-database-connectivity; here's a copy with a few tweaks (changing large fields from VARCHAR to TEXT to avoid size limits).
Bookworm (pdns 4.7):
CREATE TABLE domains ( id INT AUTO_INCREMENT, name VARCHAR(255) NOT NULL, master VARCHAR(128) DEFAULT NULL, last_check INT DEFAULT NULL, type VARCHAR(8) NOT NULL, notified_serial INT UNSIGNED DEFAULT NULL, account VARCHAR(40) CHARACTER SET 'utf8' DEFAULT NULL, options VARCHAR(64000) DEFAULT NULL, catalog VARCHAR(255) DEFAULT NULL, PRIMARY KEY (id) ) Engine=InnoDB CHARACTER SET 'latin1'; CREATE UNIQUE INDEX name_index ON domains(name); CREATE INDEX catalog_idx ON domains(catalog); CREATE TABLE records ( id BIGINT AUTO_INCREMENT, domain_id INT DEFAULT NULL, name VARCHAR(255) DEFAULT NULL, type VARCHAR(10) DEFAULT NULL, content TEXT(64000) DEFAULT NULL, ttl INT DEFAULT NULL, prio INT DEFAULT NULL, disabled TINYINT(1) DEFAULT 0, ordername VARCHAR(255) BINARY DEFAULT NULL, auth TINYINT(1) DEFAULT 1, PRIMARY KEY (id) ) Engine=InnoDB CHARACTER SET 'latin1'; CREATE INDEX nametype_index ON records(name,type); CREATE INDEX domain_id ON records(domain_id); CREATE INDEX ordername ON records (ordername); CREATE TABLE supermasters ( ip VARCHAR(64) NOT NULL, nameserver VARCHAR(255) NOT NULL, account VARCHAR(40) CHARACTER SET 'utf8' NOT NULL, PRIMARY KEY (ip, nameserver) ) Engine=InnoDB CHARACTER SET 'latin1'; CREATE TABLE comments ( id INT AUTO_INCREMENT, domain_id INT NOT NULL, name VARCHAR(255) NOT NULL, type VARCHAR(10) NOT NULL, modified_at INT NOT NULL, account VARCHAR(40) CHARACTER SET 'utf8' DEFAULT NULL, comment TEXT CHARACTER SET 'utf8' NOT NULL, PRIMARY KEY (id) ) Engine=InnoDB CHARACTER SET 'latin1'; CREATE INDEX comments_name_type_idx ON comments (name, type); CREATE INDEX comments_order_idx ON comments (domain_id, modified_at); CREATE TABLE domainmetadata ( id INT AUTO_INCREMENT, domain_id INT NOT NULL, kind VARCHAR(32), content TEXT, PRIMARY KEY (id) ) Engine=InnoDB CHARACTER SET 'latin1'; CREATE INDEX domainmetadata_idx ON domainmetadata (domain_id, kind); CREATE TABLE cryptokeys ( id INT AUTO_INCREMENT, domain_id INT NOT NULL, flags INT NOT NULL, active BOOL, published BOOL DEFAULT 1, content TEXT, PRIMARY KEY(id) ) Engine=InnoDB CHARACTER SET 'latin1'; CREATE INDEX domainidindex ON cryptokeys(domain_id); CREATE TABLE tsigkeys ( id INT AUTO_INCREMENT, name VARCHAR(255), algorithm VARCHAR(50), secret VARCHAR(255), PRIMARY KEY (id) ) Engine=InnoDB CHARACTER SET 'latin1'; CREATE UNIQUE INDEX namealgoindex ON tsigkeys(name, algorithm);
Bullseye:
CREATE TABLE domains ( id INT AUTO_INCREMENT, name VARCHAR(255) NOT NULL, master VARCHAR(128) DEFAULT NULL, last_check INT DEFAULT NULL, type VARCHAR(6) NOT NULL, notified_serial INT DEFAULT NULL, account VARCHAR(40) DEFAULT NULL, PRIMARY KEY (id) ) Engine=InnoDB; CREATE UNIQUE INDEX name_index ON domains(name); CREATE TABLE records ( id BIGINT AUTO_INCREMENT, domain_id INT DEFAULT NULL, name VARCHAR(255) DEFAULT NULL, type VARCHAR(10) DEFAULT NULL, content TEXT(64000) DEFAULT NULL, ttl INT DEFAULT NULL, prio INT DEFAULT NULL, change_date INT DEFAULT NULL, disabled TINYINT(1) DEFAULT 0, ordername VARCHAR(255) BINARY DEFAULT NULL, auth TINYINT(1) DEFAULT 1, PRIMARY KEY (id) ) Engine=InnoDB; CREATE INDEX nametype_index ON records(name,type); CREATE INDEX domain_id ON records(domain_id); CREATE INDEX recordorder ON records (domain_id, ordername); CREATE TABLE supermasters ( ip VARCHAR(64) NOT NULL, nameserver VARCHAR(255) NOT NULL, account VARCHAR(40) NOT NULL, PRIMARY KEY (ip, nameserver) ) Engine=InnoDB; CREATE TABLE comments ( id INT AUTO_INCREMENT, domain_id INT NOT NULL, name VARCHAR(255) NOT NULL, type VARCHAR(10) NOT NULL, modified_at INT NOT NULL, account VARCHAR(40) NOT NULL, comment TEXT(21844) NOT NULL, PRIMARY KEY (id) ) Engine=InnoDB; CREATE INDEX comments_domain_id_idx ON comments (domain_id); CREATE INDEX comments_name_type_idx ON comments (name, type); CREATE INDEX comments_order_idx ON comments (domain_id, modified_at); CREATE TABLE domainmetadata ( id INT AUTO_INCREMENT, domain_id INT NOT NULL, kind VARCHAR(32), content TEXT, PRIMARY KEY (id) ) Engine=InnoDB; CREATE INDEX domainmetadata_idx ON domainmetadata (domain_id, kind); CREATE TABLE cryptokeys ( id INT AUTO_INCREMENT, domain_id INT NOT NULL, flags INT NOT NULL, active BOOL, content TEXT, PRIMARY KEY(id) ) Engine=InnoDB; CREATE INDEX domainidindex ON cryptokeys(domain_id); CREATE TABLE tsigkeys ( id INT AUTO_INCREMENT, name VARCHAR(255), algorithm VARCHAR(50), secret VARCHAR(255), PRIMARY KEY (id) ) Engine=InnoDB; CREATE UNIQUE INDEX namealgoindex ON tsigkeys(name, algorithm);
Now we're ready to restart all designate services (and pdns) and see what's broken. Rebooting is easiest!
Now pdns will probably be complaining (in the syslog) about getting axfr requests for unknown domains. That's because designate doesn't automatically populate domains in pdns, it only creates them as needed when records are added. We have too many domains to wait, so the best way forward is to dump the 'domains' table from an existing node and import it on the new node.
# # on existing, working node: # mariadb-dump pdns domains > domains.sql # # on new node # mariadb pdns < domains.sql
Finally: designate-sink needs to be able to talk to the nova proxy, in order to clean up proxies associated with a deleted VM. Add access to port 5668 for the IP of the new node to project-proxy's security group. Similarly, when creating a new node make sure it can talk to the cloud puppetmasters on port 8101 so it can clean up the certs associated with a deleted VM.
Additional consideration: The galera DB (cloudcontrols) need a DB GRANT for haproxy nodes (cloudlb) for them to access designate tables. This is because some designate services (maybe designate-central in cloudservices boxes) want to talk to the galera DB and they do via haproxy (cloudlb).
This GRANT takes this form:
# for cloudlb1001.private.eqiad.wikimedia.cloud
GRANT ALL PRIVILEGES ON designate.* TO 'designate'@'172.20.1.2' IDENTIFIED BY '<redacted>';
# for cloudlb1002.private.eqiad.wikimedia.cloud
GRANT ALL PRIVILEGES ON designate.* TO 'designate'@'172.20.2.2' IDENTIFIED BY '<redacted>';
If all is well there should be a ready-made script on cloudcontrols to set up these grants: /etc/designate/designate_grants.mysql
[..] </syntaxhighlight>
Basic operations
New PTR record
Example command to create a new PTR record:
# openstack --os-project-id cloudinfra-codfw1dev recordset create --type PTR --record test.codfw1dev.wmcloud.org. --description "test" 0-29.57.15.185.in-addr.arpa. 1.0-29.57.15.185.in-addr.arpa.
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| action | CREATE |
| created_at | 2020-05-25T17:07:34.000000 |
| description | test |
| id | 450a83c9-cca2-4c82-8a28-13cf035c7702 |
| name | 1.0-29.57.15.185.in-addr.arpa. |
| project_id | cloudinfra-codfw1dev |
| records | test.codfw1dev.wmcloud.org. |
| status | PENDING |
| ttl | None |
| type | PTR |
| updated_at | None |
| version | 1 |
| zone_id | 7796b60e-7bc6-4cb4-9e4d-dae039d3f912 |
| zone_name | 0-29.57.15.185.in-addr.arpa. |
+-------------+--------------------------------------+
In previous openstack versions we were using the designate CLI, but as of Rocky this probably no longer works:
root@cloudcontrol1003:~# designate --all-tenants --os-project-name admin record-create 16.172.in-addr.arpa. \
> --name 1.0.16.172.in-addr.arpa. --type PTR --data cloudinstances2b-gw.svc.eqiad.wmflabs. \
> --description "Neutron virtual router. Record created by hand"
+-------------+------------------------------------------------+
| Field | Value |
+-------------+------------------------------------------------+
| description | Neutron virtual router. Record created by hand |
| type | PTR |
| created_at | 2018-11-30T13:26:47.000000 |
| updated_at | None |
| domain_id | 6990e139-49e6-466c-9421-46cf45f05842 |
| priority | None |
| ttl | None |
| data | cloudinstances2b-gw.svc.eqiad.wmflabs. |
| id | 855a3d4d-4caf-4652-b897-d6559a64bb4e |
| name | 1.0.16.172.in-addr.arpa. |
+-------------+------------------------------------------------+
New A records
Example command to create a new A record:
root@cloudcontrol1003:~# OS_PROJECT_ID=tools openstack recordset create --type A --records 172.16.0.24 --sudo-project-id tools svc.tools.eqiad1.wikimedia.cloud. elasticsearch
+-------------+-------------------------------------------------+
| Field | Value |
+-------------+-------------------------------------------------+
| action | CREATE |
| created_at | 2020-03-03T18:15:41.000000 |
| description | None |
| id | 35d2a87c-d355-4805-90d9-4a088f5981bc |
| name | elasticsearch.svc.tools.eqiad1.wikimedia.cloud. |
| project_id | tools |
| records | 172.16.0.24 |
| status | PENDING |
| ttl | None |
| type | A |
| updated_at | None |
| version | 1 |
| zone_id | 808263bd-522d-42c3-88a1-40af24b2929e |
| zone_name | svc.tools.eqiad1.wikimedia.cloud. |
+-------------+-------------------------------------------------+
New NS record
Example command for creating a new NS record by hand (sub domain delegation):
root@cloudcontrol1004:~# designate --all-tenants --os-project-name cloudinfra record-create wmcloud.org. --name codfw1dev.wmcloud.org --type NS --data ns0.openstack.codfw1dev.wikimediacloud.org --description "delegation to Designate in the other deployment. Record created by hand"
+-------------+-------------------------------------------------------------------------+
| Field | Value |
+-------------+-------------------------------------------------------------------------+
| description | delegation to Designate in the other deployment. Record created by hand |
| type | NS |
| created_at | 2020-01-28T10:02:28.000000 |
| updated_at | None |
| domain_id | a5f22422-815c-43d2-95aa-8fbdee95de01 |
| priority | None |
| ttl | None |
| data | ns0.openstack.codfw1dev.wikimediacloud.org. |
| id | a0a86d45-6bab-445f-8bd2-f0bf2f76e45a |
| name | codfw1dev.wmcloud.org. |
+-------------+-------------------------------------------------------------------------+
Delegations within the same openstack deployment are usually made using the wmcs-makedomain script.
Updating an existing record
Example command for updating an existing A record:
# Update A record: # designate --all-tenants --os-project-name admin record-update --name tools.db.svc.eqiad.wmflabs. --type A --data 185.15.56.54 --description "ToolsDB server. Record created by hand" df88fcb3-fbc2-42f1-bb12-2424c8b7117e 522952da-b43b-4032-bcb5-df04b6a1cdbc # which means (uids translated): # designate --all-tenants --os-project-name admin record-update --name tools.db.svc.eqiad.wmflabs. --type A --data 185.15.56.54 --description "ToolsDB server. Record created by hand" db.svc.eqiad.wmflabs. tools.db.svc.eqiad.wmflabs.
TODO: verify me and use proper syntax highlighting.
Maintenance operations
If things are misbehaving
Restart everything
The bits of designate do their best to remain synchronized and organized, but it's complicated with some races and leaks. If things are misbehaving, the first step is always to just restart everything.
We have a cookbook for that!
sudo cookbook wmcs.openstack.restart_openstack --designate --cluster-name eqiad1
Check the designate-sink logs
If new VMs aren't getting their A records (usually first noticed via a failure in the fullstack tests) and restarting everything didn't help, look in /var/log/designate/designate-sink.log to see what's happening. Normal operations like creation and deletion will show up there as warnings, just so we can tell what's going on.
Look at pdns
If all else fails and new zones aren't handled properly, there may be an issue with AXFR or the PDNS api. PDNS logs to syslog on the cloudservices nodes; you can see its logs with
tail -f /var/log/syslog | grep -i pdns
Cleaning up leaked records
In some cases, leaked DNS records may happen. See nova-fullstack for information on the VM testing that is done every 5 minutes. If you see a lot of fullstackd* DNS entries on the cloudinfra project in horizon, DNS records have leaked.
We have a custom script to detect/correct them: wmcs-dnsleaks. If this script is run with the --delete argument, it will delete leaked records, which is useful if there are many of them.
andrew@cloudcontrol2001-dev:~$ sudo wmcs-dnsleaks
checking zone: 16.172.in-addr.arpa.
Found 2 ptr recordsets for the same VM: canary2002-dev-2.cloudvirt-canary.codfw1dev.wikimedia.cloud. ['38.128.16.172.in-addr.arpa.', '170.128.16.172.in-addr.arpa.']
Found 2 ptr recordsets for the same VM: canary2003-dev-2.cloudvirt-canary.codfw1dev.wikimedia.cloud. ['134.128.16.172.in-addr.arpa.', '131.128.16.172.in-addr.arpa.']
checking zone: codfw1dev.wikimedia.cloud.
skipping public zone: codfw1dev.wmcloud.org.
skipping public zone: cloudinfra-codfw1dev.codfw1dev.wmcloud.org.
checking zone: 57.15.185.in-addr.arpa.
checking zone: 0-29.57.15.185.in-addr.arpa.
Manual fix for multiple IP records
wmcs-dnsleaks will print a message like this when it finds an A record with multiple IPs:
A record for canary1038-01.cloudvirt-canary.eqiad.wmflabs. has multiple IPs: ['172.16.2.37', '172.16.2.54'] This needs cleanup but that isn't implemented and almost never happens.
These can be manually fixed:
$ ssh cloudcontrol1003.wikimedia.org
$ sudo su -
$ source /root/novaenv.sh
$ openstack recordset set eqiad.wmflabs. canary1038-01.cloudvirt-canary.eqiad.wmflabs. --record 172.16.2.54 --edit-managed --sudo-project-id noauth-project
url: /zones/114f1333-c2c1-44d3-beb4-ebed1a91742b/recordsets/0464e35f-92a0-46f5-a31e-b5dde9fb5ea7
kwargs: {'data': '{"records": ["172.16.2.54"]}'}
+-------------+-----------------------------------------------+
| Field | Value |
+-------------+-----------------------------------------------+
| action | UPDATE |
| created_at | 2020-07-31T22:43:37.000000 |
| description | None |
| id | 0464e35f-92a0-46f5-a31e-b5dde9fb5ea7 |
| name | canary1038-01.cloudvirt-canary.eqiad.wmflabs. |
| project_id | noauth-project |
| records | 172.16.2.54 |
| status | PENDING |
| ttl | None |
| type | A |
| updated_at | 2020-08-28T21:05:51.000000 |
| version | 2 |
| zone_id | 114f1333-c2c1-44d3-beb4-ebed1a91742b |
| zone_name | eqiad.wmflabs. |
+-------------+-----------------------------------------------+
DNS zone creating / transfers
If the zone you want to create doesn't have a parent zone already registered in Designate (i.e, is not a sub-zone), you can create the zone directly in the project it will live on:
# openstack --os-project-id cloudinfra-codfw1dev zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description "floating IPs subnet" 0-29.57.15.185.in-addr.arpa.
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| action | CREATE |
| attributes | |
| created_at | 2020-05-25T16:34:38.000000 |
| description | floating IPs subnet |
| email | root@wmflabs.org |
| id | 7796b60e-7bc6-4cb4-9e4d-dae039d3f912 |
| masters | |
| name | 0-29.57.15.185.in-addr.arpa. |
| pool_id | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id | cloudinfra-codfw1dev |
| serial | 1590424478 |
| status | PENDING |
| transferred_at | None |
| ttl | 3600 |
| type | PRIMARY |
| updated_at | None |
| version | 1 |
+----------------+--------------------------------------+
But anyway, according to our DNS setup, certain domains (zones in Designate) should be created in the admin project and then transfered / relocated to the cloudinfra project.
This can be done with the openstack native transfer mechanism.
# openstack zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description "floating IPs subnet" 57.15.185.in-addr.arpa.
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| action | CREATE |
| attributes | |
| created_at | 2020-04-06T17:38:44.000000 |
| description | floating IPs subnet |
| email | root@wmflabs.org |
| id | 24e82029-c970-4422-875f-e81d9e59e33a |
| masters | |
| name | 57.15.185.in-addr.arpa. |
| pool_id | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id | admin |
| serial | 1586194724 |
| status | PENDING |
| transferred_at | None |
| ttl | 3600 |
| type | PRIMARY |
| updated_at | None |
| version | 1 |
+----------------+--------------------------------------+
# openstack zone transfer request create --target-project-id cloudinfra 67603ef4-3d64-40d6-90d3-5b7776a99034
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2020-02-18T15:35:43.000000 |
| description | None |
| id | 0a30d1fe-51ea-4da4-8a27-80c16e681ff6 |
| key | LOYRJY7Q |
| links | {u'self': u'http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones/tasks/transfer_requests/0a30d1fe-51ea-4da4-8a27-80c16e681ff6'} |
| project_id | admin |
| status | ACTIVE |
| target_project_id | cloudinfra |
| updated_at | None |
| zone_id | 67603ef4-3d64-40d6-90d3-5b7776a99034 |
| zone_name | eqiad1.wikimedia.cloud. |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+
# OS_PROJECT_ID=cloudinfra openstack zone transfer accept request --key LOYRJY7Q --transfer-id 0a30d1fe-51ea-4da4-8a27-80c16e681ff6
[..]
# OS_PROJECT_ID=cloudinfra openstack zone show 67603ef4-3d64-40d6-90d3-5b7776a99034
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| action | NONE |
| attributes | {} |
| created_at | 2019-10-18T15:57:51.000000 |
| description | None |
| email | root@wmflabs.org |
| id | 67603ef4-3d64-40d6-90d3-5b7776a99034 |
| masters | |
| name | eqiad1.wikimedia.cloud. |
| pool_id | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id | cloudinfra |
| serial | 1581676303 |
| status | ACTIVE |
| transferred_at | None |
| ttl | 3600 |
| type | PRIMARY |
| updated_at | 2020-02-18T15:38:17.000000 |
| version | 6 |
+----------------+--------------------------------------+
See phabricator T245494 - CloudVPS: figure out DNS zone ownership transfers and setup for context on how this was figured out.
Zones
tools.eqiad.wmflabs
Some special hosts are defined in profile::openstack::main::pdns::recursor_aliaser_extra_records:
- tools-db.tools.eqiad.wmflabs
- tools-redis.tools.eqiad.wmflabs
- tools-redis.eqiad.wmflabs
- puppet
.wmflabs and .wikimedia.cloud
Records created by designate-sink are marked as 'managed' which means they can not be edited with Horizon or normal command line actions. This can be overridden with the --edit-managed flag, for example:
# OS_PROJECT_ID=cloudinfra openstack recordset set eqiad1.wikimedia.cloud. tools-redis-1003.tools.eqiad1.wikimedia.cloud. --record 172.16.1.107 --edit-managed
Records under .wmflabs (and a few other things including the ptr zone for internal IPs, 16.172.in-addr.arpa) are in a special visible-only-to-Designate tenant, 'noauth-project'. Horizon and many command-line tools are unable to access that tenant, but you can see them with the 'novaadmin' user and the --sudo-project-id flag:
# openstack zone list --sudo-project-id noauth-project
Records under .wikimedia.cloud are owned by the 'cloudinfra' project and can be viewed there. They are also managed, though, so modifying them requires use of the commandline and --edit-managed, as above.
svc zone in .wikimedia.cloud domain
To create and delegate the svc.NEWPROJECT.eqiad1.wikimedia.cloud domain, do the following. Please note that the parent domain always belong to the cloudinfra project per our setup.
- Create the DNS zone for newproject in the cloudinfra project:
user@cloudcontrol1004:~ $ sudo wmcs-openstack --os-project-id cloudinfra zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 svc.newproject.eqiad1.wikimedia.cloud.
+----------------+---------------------------------------+
| Field | Value |
+----------------+---------------------------------------+
| action | CREATE |
| attributes | |
| created_at | 2020-05-21T12:27:05.000000 |
| description | None |
| email | root@wmflabs.org |
| id | 345c1fbc-4f15-4ab5-b87e-38bd284b55b9 |
| masters | |
| name | svc.newproject.eqiad1.wikimedia.cloud.|
| pool_id | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id | cloudinfra |
| serial | 1590064025 |
| status | PENDING |
| transferred_at | None |
| ttl | 3600 |
| type | PRIMARY |
| updated_at | None |
| version | 1 |
+----------------+---------------------------------------+
- create the zone transfer request, mind the project specifications:
user@cloudcontrol1004:~ $ sudo wmcs-openstack --os-project-id cloudinfra zone transfer request create --target-project-id newproject 345c1fbc-4f15-4ab5-b87e-38bd284b55b9
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2020-05-21T12:28:39.000000 |
| description | None |
| id | 3910d76b-b660-42a4-b995-21db53e3e42b |
| key | KQBAHXJ4 |
| links | {'self': 'http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones/tasks/transfer_requests/3910d76b-b660-42a4-b995-21db53e3e42b'} |
| project_id | cloudinfra |
| status | ACTIVE |
| target_project_id | paws |
| updated_at | None |
| zone_id | 345c1fbc-4f15-4ab5-b87e-38bd284b55b9 |
| zone_name | svc.newproject.eqiad1.wikimedia.cloud. |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------+
- accept the zone transfer request, again mind the project specifications:
user@cloudcontrol1004:~ $ sudo wmcs-openstack --os-project-id newproject zone transfer accept request --key G3VR0YQR --transfer-id 5d3367a7-2d3c-4ed7-8e44-8107ee9d7520
[...]
- done! confirm the new ownership:
user@cloudcontrol1004:~ $ sudo wmcs-openstack zone show 345c1fbc-4f15-4ab5-b87e-38bd284b55b9 --os-project-id paws
+----------------+----------------------------------------+
| Field | Value |
+----------------+----------------------------------------+
| action | NONE |
| attributes | |
| created_at | 2020-05-21T12:27:05.000000 |
| description | None |
| email | root@wmflabs.org |
| id | 345c1fbc-4f15-4ab5-b87e-38bd284b55b9 |
| masters | |
| name | svc.newproject.eqiad1.wikimedia.cloud. |
| pool_id | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id | newproject |
| serial | 1590064025 |
| status | ACTIVE |
| transferred_at | None |
| ttl | 3600 |
| type | PRIMARY |
| updated_at | 2020-05-21T12:32:45.000000 |
| version | 4 |
+----------------+----------------------------------------+
The instructions here are specific to this use case, including details on the ownership dancing, rather than the generic instructions in the DNS zone creating / transfers section.
Administrative scripts
See CloudVPS maintenance, DNS admin scripts.
Wiki Replica DNS
See Portal:Data_Services/Admin/Wiki_Replica_DNS
labs-ip-alias-dump
This script actually generates a lua script which is read by pdns-recursor. This is not technical anything related to Designate, but it is relevant to Cloud VPS DNS lookups. The script hooks two lua resolver hooks:
- postresolve
- Maps public IPs to private IPs so that a lookup that results in a floating IP instead returns the private IP of the instance that it is attached to.
- preresolve
- Injects additional DNS results that are only visible via the recursor. Currently used to provide DNS responses for:
- puppet.
- tools-db.tools.eqiad.wmflabs.
- tools-redis.eqiad.wmflabs.
- tools-redis.tools.eqiad.wmflabs.
The script runs via cron every 60 minutes on each pdns-recursor host.
dns-floating-ip-updater
Labs has a block of public IPs: 208.80.155.128/25
Unfortunately 208.80.155.0/25 are production IPs
Therefore production can't delegate the whole of the 155.80.208.in-addr.arpa. zone to labs NS servers for us to automate reverse DNS records from
Our workaround is RFC 2317-style delegation: We have a zone called 128-25.155.80.208.in-addr.arpa. (living under the wmflabsdotorg tenant IIRC) that production CNAMEs all the usual in-addr records for labs into, and delegates the zone to labs nameservers.
We have a script called dns-floating-ip-updater that sets up instance-{instancename}.{project}.wmflabs.org records (assuming {project}.wmflabs.org exists as a domain, if not they don't get the feature) and the reverse DNS record for that, then also sets up the reverse DNS for all manually curated A records.
What happens when someone makes a reverse DNS lookup, for let's say 208.80.155.131:
- It gets turned into a PTR lookup on
131.155.80.208.in-addr.arpa.as normal - Prod NS servers control the
155.80.208.in-addr.arpa.zone - if you open the file for that in the operations/dns.git repository ('templates' directory), you'll see the first 128 addresses are reserved for prod (some in use, some not) - So under 208.80.155, number 131 is
IN CNAME 131.128-25i.e.131.128-25.155.80.208.in-addr.arpa. 128-25.155.80.208.in-addr.arpa.is delegated from prod to labs NS, i.e. designate.- So the full lookup becomes
131.128-25.155.80.208.in-addr.arpa.and is sent to the labs NS server - This script ensures that designate knows how to answer that query, by populating all the records.
For example:
$ host bastion.wmflabs.org
bastion.wmflabs.org has address 208.80.155.129 # this is answered by the labs nameservers (designate) as they control wmflabs.org, under the wmflabsdotorg tenant
$ host 208.80.155.129
129.155.80.208.in-addr.arpa is an alias for 129.128-25.155.80.208.in-addr.arpa. # this is the production nameservers delegating the query to labs designate
129.128-25.155.80.208.in-addr.arpa domain name pointer bastion.wmflabs.org. # this is from the labs nameservers (designate) and is because bastion.wmflabs.org points to that IP
129.128-25.155.80.208.in-addr.arpa domain name pointer instance-bastion-01.bastion.wmflabs.org. # this comes from the labs nameservers (designate) and is used to indicate which instance a given IP points to, partially in case it lacks other names, but it's helpful regardless. (think what happens if we didn't have this entry but you wanted to go look up the instance on the project's instance list knowing only it's IP - and that's assuming you know which project to look under)
129.128-25.155.80.208.in-addr.arpa domain name pointer primary.bastion.wmflabs.org. # this is from the labs nameservers (designate) and is because primary.bastion.wmflabs.org points to that IP
$ host instance-bastion-01.bastion.wmflabs.org
instance-bastion-01.bastion.wmflabs.org has address 208.80.155.129 # this comes from the labs nameservers (designate) - wouldn't make sense to provide the PTR above without this
We will need to do some work on our script to be able to add new domains or public IP ranges (e.g. for other datacentres, to introduce IPv6, or to simply get more addresses).
Announcement post: https://www.mail-archive.com/labs-l@lists.wikimedia.org/msg04516.html
See also
- Portal:Cloud_VPS/Admin/OpenTofu -- manage openstack DNS (designate) via opentofu