Wikimedia Cloud Services team/EnhancementProposals/DNS domain usage
Years after the rebranding project we are still using domains with labs in them: *.wmflabs.org, *.eqiad.wmflabs. The WMCS team would like to design and begin implementing a sane replacement for these legacy domains both to remove this vestige of the former project name and to help make separations of various zones of operation more clear.
Problem statement
Things to name
- Physical hardware: cloud*.{codfw,eqiad}.wmnet, cloud*.wikimedia.org
- Cloud VPS instances: *.{codfw,eqiad}.wmflabs
- Public HTTPS proxies: *.wmflabs.org
- Toolforge webservices: tools.wmflabs.org
- Virtual services: *.svc.{codfw,eqiad}.wmflabs
We have two different DNS systems to work with as well, the production gdnsd system and the Designate/PDNS system inside Cloud VPS.
Candidate domains
Domain names we already own related to Cloud Services:
- wikimediacloudservices.org
- wikimediacloud.org
- wmcloud.org
- wikimedia.cloud
- toolforge.org
- pseudo TLDs we can resolve locally (for example *.{codfw,eqiad}.wmcloud or *.{codfw,eqiad}.wmcs)
Resolution
The #SRE offsite proposal has been chosen as the "winning" proposal.
DNS naming
- *.wikimediacloud.org
- Naming for openstack API endpoints and other shared service endpoints like elasticsearch. Subdomains may be added as needed to differentiate services and service regions/zones/clusters.
- DNS will be managed in the main Wikimedia DNS servers. Subdomains may be delegated to OpenStack Designate when appropriate.
- Examples:
- openstack.eqiad1.wikimediacloud.org
- cloudinstances2b-gw.openstack.codfw1dev.wikimediacloud.org
- nat.openstack.eqiad1.wikimediacloud.org
- *.wmcloud.org
- Naming for public exposure of Cloud VPS instances either directly (public IPv4/IPv6) or via proxy (HTTPS proxies).
- Replaces deprecated *.wmflabs.org
- DNS will be managed via OpenStack Designate in eqiad1. Subdomains may be delegated to other deployments (for example codfw1dev) as needed.
- Examples:
- bastion.wmcloud.org
- space.wmcloud.org
- outreachdashboard.wmcloud.org
- bastion.tools.wmcloud.org
- *.{deployment}.wikimedia.cloud
- Naming for internal Cloud VPS instances and services. Intended for private IP addresses.
- Replaces deprecated *.{data center}.wmflabs
- DNS will be managed via OpenStack Designate in the associated deployment.
- Examples:
- bastion-01.bastion.eqiad1.wikimedia.cloud
- k8s-control.svc.tools.wikimedia.cloud
- enwiki.analytics.db.svc.wikimedia.cloud
- *.toolforge.org
- Naming for public exposure of Toolforge tools and other 'native' Toolforge services
- Replaces deprecated tools.wmflabs.org
- DNS will be managed via OpenStack Designate.
- Examples:
- admin.toolforge.org
- openstack-browser.toolforge.org
HA / Load balancing
Production LVS servers MUST NOT be used for *.wikimediacloud.org services. Alternate load balancing systems will be designed and implemented for HA of Cloud Services core services.
See Wikimedia Cloud Services team/EnhancementProposals/OpenStackHA for further design discussion.
Proposals
Arturo
Proposal raised by Arturo Borrero Gonzalez in phab:T223902#5210949:
- Toolforge tools and other stuff native to Toolforge: we plan to use toolforge.org for them, as stated in the original docs
- cloudVPS VMs and other virtual stuff managed by openstack: we plan to use wmcloud.org for them, as stated in the original docs. This domain will be entirely managed by Designate.
- databases, like wikireplicas, toolsdb, etc. They are (or will be soon) all cloudVPS VMs. So they are covered by wmcloud.org already.
- for Openstack endpoints (the original issue in this tickets) I think we have 2 options:
- we could use $subdomain.wmcloud.org if this subdomain is not hosted by desginate (to avoid chicken-egg problems)
- we could use wikimedia.cloud, host this domain in prod DNS and dedicate it to services only. Examples: keystone.eqiad1.wikimedia.cloud, nova.codfw1dev.wikimedia.org
- supporting services, like NFS or Ceph. We are in the middle of doing a PoC for this. It's not clear what the final networking topology/layout for this would be. We could probably either:
- reuse standard prod domains (be it .wmnet or .wikimedia.org)
- try to introduce them to wikimedia.cloud as well: cloudosd1001.wikimedia.cloud? Perhaps this will be the first case of real HW being assigned anything other than .wmnet|.wikimedia.org.
Brandon
Proposal raised by BBlack in phab:T223902#5211181:
- wmcloud.org - replaces wmflabs.org, hosted directly on Designate, covers everything that's inside instances
- wikimediacloud.org - Used for the official public API service (nevermind that they're not yet fully public, initially) - The ~5 API endpoints in this ticket, and the DNS is hosted by prod DNS and not designate.
- wikimedia.cloud - Used for supporting services (NFS, Ceph, etc) or any other semi-internal usage that's sitting in some bridging position between prod and wmcs. Probably hosted by prod DNS, but could delegate subdomains to Designate if appropriate.
SRE offsite proposal
Proposal raised by Brandon, Jason, Jonh, Andrew, Bryan and Arturo @ Dublin SRE offsite Jun 2019:
- openstack.eqiad1.wikimediacloud.org --- for openstack endpoints, and other endpoints like elasticsearch, using different subdomain.
- wmcloud.org --- replaces wmflabs.org
- wikimedia.cloud --- replaces .wmflabs
- toolforge.org --- replaces tools.wmflabs.org
HA proxy in cloudcontrol servers instead of using prod LVS servers.