Jump to content

Portal:Cloud VPS/Admin/IPv6/initial deploy

From Wikitech
Tracked in Phabricator
Task T37947

This page tracks project information for the initial IPv6 deployment in Cloud VPS.

Implementation details

Some implementation details.

user facing options

The IPv6 initial deploy is paired with the VXLAN migration. For some time, there will be an option in horizon to create virtual machines attached to 3 different kind of networks:

  • VLAN/legacy
  • VXLAN/IPv4-only (a 1:1 replacement of the old legacy VLAN, but based on VXLAN)
  • VXLAN/IPv6-dualstack (VMs will get 1 IPv4 and 1 IPv6)

As the migration progresses, the options available in Horizon will get trimmed down, until everyone is on VXLAN/IPv6-dualstack.

See also Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T377467_cloud_vps_vxlan_ipv6_migration

network topology

For VMs in VXLAN/IPv6-dualstack, neutron will allocate an IPv4 and an IPv6.

  • The IPv4 is private as usual. It will get the usual egress NAT, can get associated a floating IP, etc.
  • The IPv6 is public and global. There are no NAT or floating IP options. The VM can contact the internet (and WMF wikis) with it, freely.

The edge network routing is dual-stacked. Network path for IPv4 and IPv6 is similar, barring NATs.

We have a primary neutron virtual router, then cloudgw router, then cloudsw, then the Wikimedia core routers. This is as usual, just with IPv6 support, and with no NATs in case of IPv6.

To clarify, IPv6 traffic on a VM:

  • egress: open by default, to other VMs in the same project, other VMs in other projects, the WMF networks and the internet.
  • ingress: restricted in default security group to VMs in the same project. Manual change in security groups can be made to allow ingress from VMs in other projects, the WMF networks and the internet.

firewalling

There are some special semantics here.

  • IPv6 egress traffic is allowed by default for VMs with IPv6.
  • IPv6 ingress traffic is controlled purely by local security groups on each VM. Additional firewalling can be done inside the VM via whatever method.
  • There are options for additional perimeter firewalling on cloudgw nodes, similar to IPv4, but is not active unless there is need to.

DNS

If using VXLAN/IPv6-dualstack, each VM will get a FQDN with two DNS recods, for example some-vm.some-project.eqiad1.wikimedia.cloud will get:

  • A 172.16.x.y its IPv4, as usual
  • AAAA 2a02:ec80:a000:1::x:y:z its IPv6

Additionally, using https://gitlab.wikimedia.org/repos/cloud/cloud-vps/designate-sink-wmcs-nova-fixed-ptr we will create two PTR records for each IPv4/IPv6.

addressing plan

Remember, source of truth is Netbox.

IPv6

In codfw:

In eqiad:

IPv4

In codfw:

In eqiad:

other details

Some additional information.

  • openstack APIs: can only work on IPv4 at the moment. We will need to enable IPv6 for it, see also phab:T379282.
  • nova-proxy: can only work on IPv4 at the moment. We will need to enable IPv6 for it, see also phab:T379175.
  • toolforge: we will need to figure out a number of things before introducing IPv6 inside Kubernetes, see also phab:T380060.
  • tenant networks: we have plenty of options available to us at the moment. But it was decided that this was out of the scope for this migration.

Log

Log of relevant events related to this project, including decisions, or others.

Date Event Who Additional comments
2024-11-25 Attempt to enable VXLAN/IPv6 on eqiad1 failed. There was a rollback. Arturo
2024-11-22 User-facing migration documentation bootstrapped Arturo
2024-11-21 Migration start date decided to be on 2025-01-06 WMCS WMF policy to don't announce new projects during December.
2024-11-07 Decided to keep 1500 MTU for the physical network Cathal & Arturo tricky to change -- not causing any trouble today
2024-11-06 Decision record - How to do the Cloud VPS VXLAN/IPv6 migration WMCS
2024-10-17 Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration started WMCS
2024-10-16 IPv6 virtual machine firewalling semantics clarified WMCS based purely on neutron security groups
2024-10-11 IPv6 edge routing enabled in codfw1dev WMCS & NetOps north-south traffic
2024-10-02 Initial IPv6 support enabled in codfw1dev WMCS east-west traffic
2024-09-16 IPv6 addressing plan agreement reached WMCS & NetOps
2024-09-13 IPv6 rollout should be paired with the VXLAN migration Cathal & Arturo
2024-05-13 Initial stages of the VXLAN migration started Taavi
2024-04-24 IPv6 project declared non-stalled WMCS & NetOps Both teams declare interest in re-starting work on this "soon"
2021-08-27 IPv6 project declared stalled
2018-02-21 Initial attempt to decide on IPv6 CIDR allocations for WMCS started NetOps

See also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/IPv6/initial_deploy&oldid=2249630"