VXLAN-EVPN Network Testing - Eqiad Expansion

From Wikitech
Jump to navigation Jump to search

This page documents various tests done to validate the correct operation of networking devices in the new cage (rows E/F) in Eqiad (Equinix, Ashburn). The switches in these rows has been set up differently to those in other rows, following an updated deisgn.

The principal change in the new design is the use of per-rack Vlans/subnets where possible, with end systems connected to an overlay network built with VXLAN/EVPN. The use of VXLAN/EVPN ensures that any edge-cases requiring layer-2 adjacency between racks can be addressed, if needed. The preference is to minimize where and when Vlans need to be extended between racks, however.

Test Topology

WMF Eqiad EF Test Topo.drawio.png

Tests were carried out with the above physical topology. Purple/blue links to the CR routers use 100GBase-CWDM4 and terminate into the overlay network (vrf) at layer-3 on the switches. The green links connect devices in the VXLAN fabric together, again using 100GBase-CWDM4, and provide the underlay network transport. Finally the red links are 10G DAC connections within the racks from server to switch, and terminate into overlay Vlans on the switches.

4 Dell servers, which will ultimately be the first 3 hosts provisioned in the new cage, were commandeered temporarily to perform the testing. A vanilla debian install (from debian ISO image) was used for most of the testing.

As can be seen the full Clos topology as described in the design was not ready before launch. The Juniper QFX5120-32C Spine-layer switches were unavailable due to issues with the supplier / global chip shortage. As a result of this two of the LEAF layer devices (lsw1-e1/f1, QFX5120-48Y) were configured to temporarily operate as both access and aggregation devices. These devices have enough ports to support the 6 remaining Leaf switches installed in phase 1 of the expansion. These devices, E2-E4 and F2-F4, are shown at the bottom and are also QFX5120-48Y switches.

For most tests the temporary topology does not make a difference. It does create a situation where servers connected in separate racks can have a different number of hops to reach each other, and potentially different bandwidth available, but the logical function of the network can still be assessed. Certain tests, mainly relating to failover, need to be initiated from the switches at the bottom, so elastic1093-test was used for those.

Port configurations were varied occasionally to test different features. Where relevant, device configuration snippets are included to show the setup for a test. If not explicitly detailed it can be assumed that configuration for each test has not diverged from the previous.

Functional Tests

L2 Access port

This is a very basic test to validate the normal configuration for an access port works as expected and MAC addresses are learnt by the top-of-rack switch when unicast frames are transmitted by an end host.

Relevant config

ms-fe1012-test was placed into vlan 1031 and the link brought up:

   cmooney@lsw1-e1-eqiad> show configuration vlans private1-e1-eqiad 
   vlan-id 1031;
   l3-interface irb.1031;
   vxlan {
       vni 2001031;
   }
   cmooney@lsw1-e1-eqiad> show configuration interfaces interface-range vlan-private1-e1-eqiad 
   member xe-0/0/7;
   mtu 9192;
   unit 0 {
       family ethernet-switching {
           interface-mode access;
           vlan {
               members private1-e1-eqiad;
           }
       }
   }
   cmooney@lsw1-e1-eqiad> show configuration interfaces xe-0/0/7                                  
   description "ms-fe1012 {#5004}";
   root@ms-fe1012-test:~# ip -br link show enp101s0f0np0
   enp101s0f0np0    UP             e4:3d:1a:12:b1:00 <BROADCAST,MULTICAST,UP,LOWER_UP> 

Results

MAC address learnt on port as expected:

   cmooney@lsw1-e1-eqiad> show ethernet-switching table vlan-name private1-e1-eqiad 
   
   MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
              SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
   
   
   Ethernet switching table : 1 entries, 1 learned
   Routing instance : default-switch
      Vlan                MAC                 MAC      Logical                SVLBNH/      Active
      name                address             flags    interface              VENH Index   source
      private1-e1-eqiad   e4:3d:1a:12:b1:00   D        xe-0/0/7.0

L2 Trunk port

This is a similar basic test to validate the normal configuration for a trunk port works as expected.

Relevant config

an-worker1147 was connected to vlans 1031 (private1-e1-eqiad) and 1039 (analytics1-e1-eqiad) with a 802.1q trunk as follows:

   cmooney@lsw1-e1-eqiad> show configuration vlans analytics1-e1-eqiad         
   vlan-id 1039;
   l3-interface irb.1039;
   vxlan {
       vni 2001039;
   }
   cmooney@lsw1-e1-eqiad> show configuration vlans private1-e1-eqiad     
   vlan-id 1031;
   l3-interface irb.1031;
   vxlan {
       vni 2001031;
   }
   cmooney@lsw1-e1-eqiad> show configuration interfaces xe-0/0/6              
   description an-worker1147;
   native-vlan-id 1039;
   mtu 9192;
   unit 0 {
       family ethernet-switching {
           interface-mode trunk;
           vlan {
               members [ private1-e1-eqiad analytics1-e1-eqiad ];
           }
       }
   }
   root@an-worker1147-test:~# ip -d link show eno2np1
   5: eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP mode DEFAULT group default qlen 1000
       link/ether e4:3d:1a:54:14:45 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 60 maxmtu 9600 addrgenmode eui64 numtxqueues 74 numrxqueues 74 gso_max_size 65536 gso_max_segs 65535 portname p1 switchid 441454feff1a3de4 
       altname enp24s0f1np1
   root@an-worker1147-test:~# ip -d link show eno2np1.1031
   6: eno2np1.1031@eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP mode DEFAULT group default qlen 1000
       link/ether e4:3d:1a:54:14:45 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65535 
       vlan protocol 802.1Q id 1031 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 

Results

MAC addresses learnt on both Vlans as expected:

   cmooney@lsw1-e1-eqiad> show ethernet-switching table interface xe-0/0/6.0 | match xe-0/0/6 
   
   MAC database for interface xe-0/0/6.0
      analytics1-e1-eqiad e4:3d:1a:54:14:45   D        xe-0/0/6.0           
      private1-e1-eqiad   e4:3d:1a:54:14:45   D        xe-0/0/6.0      


Traffic on non-native vlan received by end host with tags:

   root@an-worker1147-test:~# tcpdump -i eno2np1 -l -p -nn -e port not 22 and net 10.64.0.0/10
   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   listening on eno2np1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   13:08:13.276275 a4:e1:1a:81:3a:80 > e4:3d:1a:54:14:45, ethertype 802.1Q (0x8100), length 102: vlan 1031, p 0, ethertype IPv4 (0x0800), 10.64.130.1 > 10.64.130.11: ICMP echo request, id 6677, seq 0, length 64
   13:08:13.276315 e4:3d:1a:54:14:45 > a4:e1:1a:81:3a:80, ethertype 802.1Q (0x8100), length 102: vlan 1031, p 0, ethertype IPv4 (0x0800), 10.64.130.11 > 10.64.130.1: ICMP echo reply, id 6677, seq 0, length 64
   13:08:14.276074 a4:e1:1a:81:3a:80 > e4:3d:1a:54:14:45, ethertype 802.1Q (0x8100), length 102: vlan 1031, p 0, ethertype IPv4 (0x0800), 10.64.130.1 > 10.64.130.11: ICMP echo request, id 6677, seq 1, length 64
   13:08:14.276107 e4:3d:1a:54:14:45 > a4:e1:1a:81:3a:80, ethertype 802.1Q (0x8100), length 102: vlan 1031, p 0, ethertype IPv4 (0x0800), 10.64.130.11 > 10.64.130.1: ICMP echo reply, id 6677, seq 1, length 64
   13:08:36.097539 e4:3d:1a:54:14:45 > a4:e1:1a:81:3a:80, ethertype IPv4 (0x0800), length 81: 10.64.138.11.51007 > 10.3.0.1.53: 59784+ A? 1.debian.pool.ntp.org. (39)
   13:08:36.097559 e4:3d:1a:54:14:45 > a4:e1:1a:81:3a:80, ethertype IPv4 (0x0800), length 81: 10.64.138.11.51007 > 10.3.0.1.53: 40077+ AAAA? 1.debian.pool.ntp.org. (39)

Routed port in VRF

Very simple test to confirm link-layer IP connectivity of a routed port terminated into an overlay VRF.

Relevant config

   cmooney@lsw1-e1-eqiad> show configuration interfaces et-0/0/48 unit 100                
   description "cr1-eqiad et-1/0/2.100 - Production VRF";
   vlan-id 100;
   family inet {
       address 10.66.0.9/31;
   }
   family inet6 {
       address 2620:0:861:fe07::2/64;
   }
   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION | display set | match et-0/0/48
   set routing-instances PRODUCTION interface et-0/0/48.100

Results

IPv4

   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 10.66.0.8 size 9000 do-not-fragment count 2            
   PING 10.66.0.8 (10.66.0.8): 9000 data bytes
   9008 bytes from 10.66.0.8: icmp_seq=0 ttl=64 time=2.722 ms
   9008 bytes from 10.66.0.8: icmp_seq=1 ttl=64 time=2.896 ms
   
   --- 10.66.0.8 ping statistics ---
   2 packets transmitted, 2 packets received, 0% packet loss
   round-trip min/avg/max/stddev = 2.722/2.809/2.896/0.087 ms
   

IPv6

   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 2620:0:861:fe07::1 size 9000 do-not-fragment count 2 source 2620:0:861:fe07::2 
   PING6(9048=40+8+9000 bytes) 2620:0:861:fe07::2 --> 2620:0:861:fe07::1
   9008 bytes from 2620:0:861:fe07::1, icmp_seq=0 hlim=64 time=2.915 ms
   9008 bytes from 2620:0:861:fe07::1, icmp_seq=1 hlim=64 time=2.719 ms
   
   --- 2620:0:861:fe07::1 ping6 statistics ---
   2 packets transmitted, 2 packets received, 0% packet loss
   round-trip min/avg/max/std-dev = 2.719/2.817/2.915/0.098 ms

Standard IP gateway / IRB in VRF

This test confirms routing via an IRB interface within an overlay routing instance works as expected.

Relevant Config

   cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1031 
   description private1-e1-eqiad;
   family inet {
       address 10.64.130.1/24;
   }
   family inet6 {
       address 2620:0:861:109::1/64;
   }
   root@ms-fe1012-test:~# ip -br addr show enp101s0f0np0
   enp101s0f0np0    UP             10.64.130.10/24 2620:0:861:109::10/64 fe80::e63d:1aff:fe12:b100/64 

Results

IPv4

   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 10.64.130.0    
   
   PRODUCTION.inet.0: 34 destinations, 36 routes (34 active, 0 holddown, 0 hidden)
   @ = Routing Use Only, # = Forwarding Use Only
   + = Active Route, - = Last Active, * = Both
   
   10.64.130.0/24     *[Direct/0] 5d 20:50:06
                       >  via irb.1031
   cmooney@lsw1-e1-eqiad> show arp interface irb.1031 no-resolve 
   MAC Address       Address         Interface                Flags
   e4:3d:1a:12:b1:00 10.64.130.10    irb.1031 [xe-0/0/7.0]    permanent remote
   root@ms-fe1012-test:~# ping -c 2 10.64.130.1 
   PING 10.64.130.1 (10.64.130.1) 56(84) bytes of data.
   64 bytes from 10.64.130.1: icmp_seq=1 ttl=64 time=5.55 ms
   64 bytes from 10.64.130.1: icmp_seq=2 ttl=64 time=6.90 ms
   root@ms-fe1012-test:~# ip -4 neigh show
   10.64.130.1 dev enp101s0f0np0 lladdr a4:e1:1a:81:3a:80 REACHABLE


IPv6

   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 2620:0:861:fe07:: 
   
   PRODUCTION.inet6.0: 37 destinations, 39 routes (37 active, 0 holddown, 0 hidden)
   + = Active Route, - = Last Active, * = Both
   
   2620:0:861:fe07::/64
                      *[Direct/0] 4d 14:45:08
                       >  via et-0/0/48.100
   cmooney@lsw1-e1-eqiad> show ipv6 neighbors interface irb.1031 
   IPv6 Address                            Linklayer Address  State       Exp   Rtr  Secure  Interface               
   2620:0:861:109::10                       e4:3d:1a:12:b1:00  reachable   0     no   no      irb.1031 [xe-0/0/7.0]   
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 2620:0:861:109::10 source 2620:0:861:109::1    
   PING6(56=40+8+8 bytes) 2620:0:861:109::1 --> 2620:0:861:109::10
   16 bytes from 2620:0:861:109::10, icmp_seq=0 hlim=64 time=0.633 ms
   16 bytes from 2620:0:861:109::10, icmp_seq=1 hlim=64 time=0.511 ms


   root@ms-fe1012-test:~# ping -c 2 2620:0:861:109::1
   PING 2620:0:861:109::1(2620:0:861:109::1) 56 data bytes
   64 bytes from 2620:0:861:109::1: icmp_seq=1 ttl=64 time=0.475 ms
   64 bytes from 2620:0:861:109::1: icmp_seq=2 ttl=64 time=0.470 ms
   root@ms-fe1012-test:~# ip -6 neigh show
   2620:0:861:109::1 dev enp101s0f0np0 lladdr a4:e1:1a:81:3a:80 STALE

Remote MAC Learning on Vlan/VNI

This test validates MAC address distribution across multiple switches, using BGP EVPN. In order to perform the test Vlan private1-f1-eqiad was added temporarily to switch LSW1-E1, and trunked out to an-worker1147-test.

Relevant Config

   cmooney@lsw1-e1-eqiad> show configuration interfaces xe-0/0/6     
   description an-worker1147;
   native-vlan-id 1039;
   mtu 9192;
   unit 0 {
       family ethernet-switching {
           interface-mode trunk;
           vlan {
               members [ private1-e1-eqiad private1-f1-eqiad analytics1-e1-eqiad ];
           }
       }
   }

Results

MAC address is learnt locally on lsw-e1 on port xe-0/0/6.0:

   cmooney@lsw1-e1-eqiad> show ethernet-switching table vlan-name private1-f1-eqiad            
   
   MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
              SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
   
   
   Ethernet switching table : 3 entries, 3 learned
   Routing instance : default-switch
      Vlan                MAC                 MAC      Logical                SVLBNH/      Active
      name                address             flags    interface              VENH Index   source
      private1-f1-eqiad   00:00:5e:11:fa:ce   DRP      esi.1797               1796         05:00:00:fd:2a:00:1e:88:8b:00 
      private1-f1-eqiad   a4:e1:1a:81:9e:80   DR       vtep.32770                          10.64.128.7                   
      private1-f1-eqiad   e4:3d:1a:54:14:45   D        xe-0/0/6.0


BGP EVPN type 2 route containing this MAC address is received on lsw-f1 as a result. MAC is learnt on VNI 200135, with protocol next-hop of lsw-e1 loopback/VTEP IP.

   cmooney@lsw1-f1-eqiad> show route protocol bgp table bgp.evpn.0 evpn-mac-address e4:3d:1a:54:14:45 extensive    
   
   2:10.64.128.3:64810::2001035::e4:3d:1a:54:14:45/304 MAC/IP (1 entry, 1 announced)
   TSI:
   Page 0 idx 0, (group EVPN_RR_CLIENTS type Internal) Type 1 val 0xe3c20a4 (adv_entry)
      Advertised metrics:
        Nexthop: 10.64.128.3
        Localpref: 100
        AS path: [64810] I
        Communities: target:64810:1035 encapsulation:vxlan(0x8)
        Cluster ID: 10.64.128.0
        Originator ID: 10.64.128.3
       Advertise: 0000003f
   Path 2:10.64.128.3:64810::2001035::e4:3d:1a:54:14:45
   from 10.64.128.3
   Vector len 4.  Val: 0
           *BGP    Preference: 170/-101
                   Route Distinguisher: 10.64.128.3:64810
                   Next hop type: Indirect, Next hop index: 0
                   Address: 0xc6864c8
                   Next-hop reference count: 109
                   Source: 10.64.128.3
                   Protocol next hop: 10.64.128.3
                   Indirect next hop: 0x2 no-forward INH Session ID: 0x0
                   State: <Active Int Ext>
                   Local AS: 64810 Peer AS: 64810
                   Age: 1:14:23 	Metric2: 8 
                   Validation State: unverified 
                   Task: BGP_64810.10.64.128.3
                   Announcement bits (1): 1-BGP_RT_Background 
                   AS path: I 
                   Communities: target:64810:1035 encapsulation:vxlan(0x8)
                   Import Accepted
                   Route Label: 2001035
                   ESI: 00:00:00:00:00:00:00:00:00:00
                   Localpref: 100
                   Router ID: 10.64.128.3
                   Secondary Tables: default-switch.evpn.0
                   Thread: junos-main 
                   Indirect next hops: 1
                           Protocol next hop: 10.64.128.3 Metric: 8
                           Indirect next hop: 0x2 no-forward INH Session ID: 0x0
                           Indirect path forwarding next hops: 1
                                   Next hop type: Router
                                   Next hop: 10.64.129.6 via et-0/0/52.0
                                   Session Id: 0x0
                                   10.64.128.3/32 Originating RIB: inet.0
                                     Metric: 8 Node path count: 1
                                     Forwarding nexthops: 1
                                           Next hop type: Router
                                           Next hop: 10.64.129.6 via et-0/0/52.0
                                           Session Id: 0x0

The MAC is placed into the Vlan forwarding table as expected:

   cmooney@lsw1-f1-eqiad> show ethernet-switching table vlan-name private1-f1-eqiad                                
   
   MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
              SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
   
   
   Ethernet switching table : 4 entries, 4 learned
   Routing instance : default-switch
      Vlan                MAC                 MAC      Logical                SVLBNH/      Active
      name                address             flags    interface              VENH Index   source
      private1-f1-eqiad   00:00:5e:11:fa:ce   DRP      esi.1780               1779         05:00:00:fd:2a:00:1e:88:8b:00 
      private1-f1-eqiad   a4:e1:1a:81:3a:80   DR       vtep.32769                          10.64.128.3                   
      private1-f1-eqiad   e4:3d:1a:54:14:45   DR       vtep.32769                          10.64.128.3
      private1-f1-eqiad   e4:3d:1a:54:ab:a7   D        xe-0/0/6.0

Client to Client L2 Unicast forwarding

These tests verify that end hosts can communicate directly amongst themselves within a particular Vlan.

Same Switch (Pure L2)

Relevant Config

   cmooney@lsw1-e1-eqiad> show configuration interfaces xe-0/0/6         
   description an-worker1147;
   native-vlan-id 1039;
   mtu 9192;
   unit 0 {
       family ethernet-switching {
           interface-mode trunk;
           vlan {
               members [ private1-e1-eqiad private1-f1-eqiad analytics1-e1-eqiad ];
           }
       }
   }
   cmooney@lsw1-e1-eqiad> show configuration interfaces interface-range vlan-private1-e1-eqiad 
   member xe-0/0/7;
   mtu 9192;
   unit 0 {
       family ethernet-switching {
           interface-mode access;
           vlan {
               members private1-e1-eqiad;
           }
       }
   }
   cmooney@lsw1-e1-eqiad> show configuration interfaces xe-0/0/7    
   description "ms-fe1012 {#5004}";
   root@ms-fe1012-test:~# ip -d link show enp101s0f0np0
   2: enp101s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP mode DEFAULT group default qlen 1000
   link/ether e4:3d:1a:12:b1:00 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 60 maxmtu 9600 addrgenmode eui64 numtxqueues 74 numrxqueues 74 gso_max_size 65536 gso_max_segs 65535 portname p0 switchid 00b112feff1a3de4 
   root@an-worker1147-test:~# ip -d link show eno2np1.1031
   6: eno2np1.1031@eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP mode DEFAULT group default qlen 1000
   link/ether e4:3d:1a:54:14:45 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65535 
   vlan protocol 802.1Q id 1031 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 


Results

   cmooney@lsw1-e1-eqiad> show ethernet-switching table vlan-name private1-e1-eqiad 
   
   MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
              SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
   
   
   Ethernet switching table : 2 entries, 2 learned
   Routing instance : default-switch
      Vlan                MAC                 MAC      Logical                SVLBNH/      Active
      name                address             flags    interface              VENH Index   source
      private1-e1-eqiad   e4:3d:1a:12:b1:00   D        xe-0/0/7.0           
      private1-e1-eqiad   e4:3d:1a:54:14:45   D        xe-0/0/6.0   


   root@an-worker1147-test:~# ping -c 2 10.64.130.10
   PING 10.64.130.10 (10.64.130.10) 56(84) bytes of data.
   64 bytes from 10.64.130.10: icmp_seq=1 ttl=64 time=0.172 ms
   64 bytes from 10.64.130.10: icmp_seq=2 ttl=64 time=0.209 ms
   root@ms-fe1012-test:~# tcpdump -i enp101s0f0np0 -l -p -nn -e icmp
   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   listening on enp101s0f0np0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   20:41:55.359338 e4:3d:1a:54:14:45 > e4:3d:1a:12:b1:00, ethertype IPv4 (0x0800), length 98: 10.64.130.11 > 10.64.130.10: ICMP echo request, id 54201, seq 20, length 64
   20:41:55.359373 e4:3d:1a:12:b1:00 > e4:3d:1a:54:14:45, ethertype IPv4 (0x0800), length 98: 10.64.130.10 > 10.64.130.11: ICMP echo reply, id 54201, seq 20, length 64


Remote Switch (VXLAN tunneled)

Relevant Config

Same as for test 2.5: Remote MAC Learning on Vlan/VNI

Results

Same MAC tables as for test 2.5: Remote MAC Learning on Vlan/VNI

   root@an-worker1148-test:~# ping 10.64.134.72 
   PING 10.64.134.72 (10.64.134.72) 56(84) bytes of data.
   64 bytes from 10.64.134.72: icmp_seq=1 ttl=64 time=19.3 ms
   64 bytes from 10.64.134.72: icmp_seq=2 ttl=64 time=0.180 ms
   root@an-worker1147-test:~# tcpdump -i eno2np1.1035 -e -l -p -nn icmp
   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   listening on eno2np1.1035, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   20:48:52.448479 e4:3d:1a:54:ab:a7 > e4:3d:1a:54:14:45, ethertype IPv4 (0x0800), length 98: 10.64.134.11 > 10.64.134.72: ICMP echo request, id 62059, seq 28, length 64
   20:48:52.448505 e4:3d:1a:54:14:45 > e4:3d:1a:54:ab:a7, ethertype IPv4 (0x0800), length 98: 10.64.134.72 > 10.64.134.11: ICMP echo reply, id 62059, seq 28, length 64

Client to Client broadcast forwarding / ingress replication

These test verify broadcasts are correctly flooded to all ports in the Vlan, both local or remote.

Same Switch (Pure L2)

Test involves generating a broadcast on ms-fe1012-test into Vlan1031 / private-e1-eqiad, and confirm it is received on an-worker1147-test. Both of these hosts are connected to lsw-e1-eqiad, so the frame will be processed as a regular L2 frame between local ports in the same Vlan.

Transmit side:

   root@ms-fe1012-test:~# echo "test broadcast in vlan 1031 private-e1-eqiad" | nc -u -b 255.255.255.255 12101 

Receive side:

   root@an-worker1147-test:~# tcpdump -X -i eno2np1.1031 -l -p -nn -e
   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   listening on eno2np1.1031, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   21:22:29.263680 e4:3d:1a:12:b1:00 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 86: 208.80.154.226.46774 > 255.255.255.255.12101: UDP, length 44
   	0x0000:  4500 0048 903c 4000 4011 3f36 d050 9ae2  E..H.<@.@.?6.P..
   	0x0010:  ffff ffff b6b6 2f45 0034 0d9a 7465 7374  ....../E.4..test
   	0x0020:  2062 726f 6164 6361 7374 2069 6e20 766c  .broadcast.in.vl
   	0x0030:  616e 2031 3033 3120 7072 6976 6174 652d  an.1031.private-
   	0x0040:  6531 2d65 7169 6164                      e1-eqiad.


Remote Switch (VXLAN tunneled)

Test involves generating a broadcast on an-worker1148, connected to lsw1-f1-eqiad, and confirming receipt on an-worker1147-test, connected to lsw-e1-eqiad. Test frame will be transmitted in Vlan1035 / private-f1-eqiad, which is configured on both switches for the purposes of testing.

Transmit side:

root@an-worker1148-test:~# echo "test broadcast in vlan 1035 private-f1-eqiad" | nc -u -b 255.255.255.255 12345

Receive side:

   root@an-worker1147-test:~# tcpdump -X -i eno2np1.1035 -l -p -nn -e
   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   listening on eno2np1.1035, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   21:36:49.189166 e4:3d:1a:54:ab:a7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 87: 10.64.134.11.48282 > 255.255.255.255.12345: UDP, length 45
   	0x0000:  4500 0049 0a29 4000 4011 a030 0a40 860b  E..I.)@.@..0.@..
   	0x0010:  ffff ffff bc9a 3039 0035 d2a7 7465 7374  ......09.5..test
   	0x0020:  2062 726f 6164 6361 7374 2069 6e20 766c  .broadcast.in.vl
   	0x0030:  616e 2031 3033 3520 7072 6976 6174 652d  an.1035.private-
   	0x0040:  6631 2d65 7169 6164 0a                   f1-eqiad.

Client to Client L2 multicast forwarding / ingress replication

Same Switch (Pure L2)

Config the same as 2.7.1 - Client to Client broadcast forwarding / ingress replication

Results

   root@ms-fe1012-test:~# echo "test multicast" | nc -q 1 -u -b 224.0.0.26 12345 
   root@an-worker1147-test:~# tcpdump -e -i eno2np1.1031 -l -nn net 224.0.0.0/8
   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   listening on eno2np1.1031, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   14:43:32.138562 e4:3d:1a:12:b1:00 > 01:00:5e:00:00:1a, ethertype IPv4 (0x0800), length 60: 10.64.130.10.59844 > 224.0.0.26.12345: UDP, length 15


Remote Switch (VXLAN tunneled)

Config the same as 2.7.2 - Client to Client broadcast forwarding / ingress replication

Results

   root@an-worker1148-test:~# echo "test multicast evpn type 3" | nc -q 1 -u -b 224.0.0.26 12345
   root@an-worker1147-test:~# tcpdump -e -i eno2np1.1035 -l -nn net 224.0.0.0/8   
   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   listening on eno2np1.1035, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   17:20:22.025485 e4:3d:1a:54:ab:a7 > 01:00:5e:00:00:1a, ethertype IPv4 (0x0800), length 69: 10.64.134.11.46904 > 224.0.0.26.12345: UDP, length 27

Anycast IP gateway / IRB across multiple switches

When Vlans are stretched across multiple devices 'anycast gws' are configured on each top of rack, with every one acting as gateway for directly connected devices on the Vlan. This test confirms that this is working and the different switches are all responding for this IP.

Config

Tests were done on Vlan1035 / private-f1-eqiad, which is connected to both an-worker1148-test (10.64.134.11) on lsw1-f1-eqiad, and an-worker1147-test (10.64.134.72) on lsw1-e1-eqiad. Layer-2 config the same as in 2.5: Remote MAC Learning on Vlan/VNI.

The IRB interface was configured as follows to provide Anycast GW support:

   cmooney@lsw1-e1-eqiad> show configuration interfaces irb unit 1035 
   virtual-gateway-accept-data;
   description private1-f1-eqiad;
   family inet {
       address 10.64.134.253/24 {
           preferred;
           virtual-gateway-address 10.64.134.1;
       }
   }
   family inet6 {
       address 2620:0:861:10d::253/64 {
           preferred;
           virtual-gateway-address 2620:0:861:10d::1;
       }
   }
   virtual-gateway-v4-mac 00:00:5e:11:fa:ce;
   virtual-gateway-v6-mac 00:00:5e:11:fa:ce;
   cmooney@lsw1-f1-eqiad> show configuration interfaces irb unit 1035 
   virtual-gateway-accept-data;
   description private1-f1-eqiad;
   family inet {
       address 10.64.134.254/24 {
           preferred;
           virtual-gateway-address 10.64.134.1;
       }
   }
   family inet6 {
       address 2620:0:861:10d::254/64 {
           preferred;
           virtual-gateway-address 2620:0:861:10d::1;
       }
   }
   virtual-gateway-v4-mac 00:00:5e:11:fa:ce;
   virtual-gateway-v6-mac 00:00:5e:11:fa:ce;


IPv4

Results

The 3 IPs (one VIP, and 2 per-switch IPs) were pinged from each an-worker1147-test and got a response from each

   root@an-worker1147-test:~# ping -c 1 10.64.134.1
   PING 10.64.134.1 (10.64.134.1) 56(84) bytes of data.
   64 bytes from 10.64.134.1: icmp_seq=1 ttl=64 time=0.677 ms
   root@an-worker1147-test:~# ping -c 1 10.64.134.253
   PING 10.64.134.253 (10.64.134.253) 56(84) bytes of data.
   64 bytes from 10.64.134.253: icmp_seq=1 ttl=64 time=7.93 ms
   root@an-worker1147-test:~# ping -c 1 10.64.134.254
   PING 10.64.134.254 (10.64.134.254) 56(84) bytes of data.
   64 bytes from 10.64.134.254: icmp_seq=1 ttl=64 time=6.06 ms


Doing a monitor on packets processed by lsw1-e1-eqiad we can see that the response from the VIP was processed by it, as well as the ping to the unicast IP:

   cmooney@lsw1-e1-eqiad> monitor traffic interface irb.1035 matching "icmp" no-resolve    
   verbose output suppressed, use <detail> or <extensive> for full protocol decode
   Address resolution is OFF.
   Listening on irb.1035, capture size 96 bytes
   
   12:25:51.348223  In IP 10.64.134.72 > 10.64.134.1: ICMP echo request, id 46233, seq 1, length 64
   12:25:51.348236 Out IP truncated-ip - 52 bytes missing! 10.64.134.1 > 10.64.134.72: ICMP echo reply, id 46233, seq 1, length 64
   12:25:57.886469  In IP 10.64.134.72 > 10.64.134.253: ICMP echo request, id 2678, seq 1, length 64
   12:25:57.886482 Out IP truncated-ip - 52 bytes missing! 10.64.134.253 > 10.64.134.72: ICMP echo reply, id 2678, seq 1, length 64


The ping to .254, unicast IP on lsw1-f1-eqiad, did not show up in that capture, but instead appears in an equivalent capture on lsw1-f1-eqiad:

   cmooney@lsw1-f1-eqiad> monitor traffic interface irb.1035 matching "icmp" no-resolve 
   verbose output suppressed, use <detail> or <extensive> for full protocol decode
   Address resolution is OFF.
   Listening on irb.1035, capture size 96 bytes
   
   12:26:00.938116  In IP 10.64.134.72 > 10.64.134.254: ICMP echo request, id 16304, seq 1, length 64


The reverse is the case when the pings are done from an-worker1148-test:

   root@an-worker1148-test:~# ping -c 1 10.64.134.1
   PING 10.64.134.1 (10.64.134.1) 56(84) bytes of data.
   64 bytes from 10.64.134.1: icmp_seq=1 ttl=64 time=14.5 ms
   root@an-worker1148-test:~# ping -c 1 10.64.134.253
   PING 10.64.134.253 (10.64.134.253) 56(84) bytes of data.
   64 bytes from 10.64.134.253: icmp_seq=1 ttl=64 time=14.9 ms
   root@an-worker1148-test:~# ping -c 1 10.64.134.254
   PING 10.64.134.254 (10.64.134.254) 56(84) bytes of data.
   64 bytes from 10.64.134.254: icmp_seq=1 ttl=64 time=6.25 ms
   cmooney@lsw1-e1-eqiad> monitor traffic interface irb.1035 matching "icmp" no-resolve    
   verbose output suppressed, use <detail> or <extensive> for full protocol decode
   Address resolution is OFF.
   Listening on irb.1035, capture size 96 bytes
   
   12:29:47.721948  In IP 10.64.134.11 > 10.64.134.253: ICMP echo request, id 52386, seq 1, length 64
   cmooney@lsw1-f1-eqiad> monitor traffic interface irb.1035 matching "icmp" no-resolve    
   verbose output suppressed, use <detail> or <extensive> for full protocol decode
   Address resolution is OFF.
   Listening on irb.1035, capture size 96 bytes
   
   12:29:42.998551  In IP 10.64.134.11 > 10.64.134.1: ICMP echo request, id 375, seq 1, length 64
   12:29:42.998564 Out IP truncated-ip - 52 bytes missing! 10.64.134.1 > 10.64.134.11: ICMP echo reply, id 375, seq 1, length 64
   12:29:50.194240  In IP 10.64.134.11 > 10.64.134.254: ICMP echo request, id 8113, seq 1, length 64
   12:29:50.194252 Out IP truncated-ip - 52 bytes missing! 10.64.134.254 > 10.64.134.11: ICMP echo reply, id 8113, seq 1, length 64


ARP entries appear as expected on both hosts, the VIP shows with the same MAC on both devices:

   root@an-worker1147-test:~# ip neigh show dev eno2np1.1035
   10.64.134.1 lladdr 00:00:5e:11:fa:ce STALE
   10.64.134.11 lladdr e4:3d:1a:54:ab:a7 STALE
   10.64.134.253 lladdr a4:e1:1a:81:3a:80 STALE
   10.64.134.254 lladdr a4:e1:1a:81:9e:80 STALE


   root@an-worker1148-test:~# ip neigh show 
   10.64.134.1 dev eno2np1 lladdr 00:00:5e:11:fa:ce STALE
   10.64.134.10 dev eno2np1 lladdr e4:3d:1a:54:14:45 STALE
   10.64.134.72 dev eno2np1 lladdr e4:3d:1a:54:14:45 STALE
   10.64.134.253 dev eno2np1 lladdr a4:e1:1a:81:3a:80 STALE
   10.64.134.254 dev eno2np1 lladdr a4:e1:1a:81:9e:80 DELAY


IPv6

Results

   root@an-worker1147-test:~# ping -c 1 2620:0:861:10d::1
   PING 2620:0:861:10d::1(2620:0:861:10d::1) 56 data bytes
   64 bytes from 2620:0:861:10d::1: icmp_seq=1 ttl=64 time=0.454 ms
   root@an-worker1147-test:~# ping -c 1 2620:0:861:10d::253
   PING 2620:0:861:10d::253(2620:0:861:10d::253) 56 data bytes
   64 bytes from 2620:0:861:10d::253: icmp_seq=1 ttl=64 time=11.1 ms
   root@an-worker1147-test:~# ping -c 1 2620:0:861:10d::254
   PING 2620:0:861:10d::254(2620:0:861:10d::254) 56 data bytes
   64 bytes from 2620:0:861:10d::254: icmp_seq=1 ttl=64 time=11.2 ms
   cmooney@lsw1-e1-eqiad> monitor traffic interface irb.1035 matching "icmp6" no-resolve     
   verbose output suppressed, use <detail> or <extensive> for full protocol decode
   Address resolution is OFF.
   Listening on irb.1035, capture size 96 bytes
   
   12:35:29.299922  In IP6 2620:0:861:10d::72 > 2620:0:861:10d::1: ICMP6, echo request, seq 1, length 64
   12:35:29.299943 Out [|ip6]
   12:35:58.340742  In IP6 2620:0:861:10d::72 > 2620:0:861:10d::253: ICMP6, echo request, seq 1, length 64
   12:35:58.340766 Out [|ip6]
   cmooney@lsw1-f1-eqiad> monitor traffic interface irb.1035 matching "icmp6" no-resolve   
   verbose output suppressed, use <detail> or <extensive> for full protocol decode
   Address resolution is OFF.
   Listening on irb.1035, capture size 96 bytes
   
   12:36:03.678690  In IP6 2620:0:861:10d::72 > 2620:0:861:10d::254: ICMP6, echo request, seq 1, length 64

The same situation is reversed, as expected, doing the same pings from the other host:

   root@an-worker1148-test:~# ping -c 1 2620:0:861:10d::1
   PING 2620:0:861:10d::1(2620:0:861:10d::1) 56 data bytes
   64 bytes from 2620:0:861:10d::1: icmp_seq=1 ttl=64 time=110 ms
   root@an-worker1148-test:~# ping -c 1 2620:0:861:10d::253
   PING 2620:0:861:10d::253(2620:0:861:10d::253) 56 data bytes
   64 bytes from 2620:0:861:10d::253: icmp_seq=1 ttl=64 time=11.3 ms
   root@an-worker1148-test:~# ping -c 1 2620:0:861:10d::254
   PING 2620:0:861:10d::254(2620:0:861:10d::254) 56 data bytes
   64 bytes from 2620:0:861:10d::254: icmp_seq=1 ttl=64 time=11.2 ms
   cmooney@lsw1-e1-eqiad> monitor traffic interface irb.1035 matching "icmp6" no-resolve    
   verbose output suppressed, use <detail> or <extensive> for full protocol decode
   Address resolution is OFF.
   Listening on irb.1035, capture size 96 bytes
   
   12:40:55.072630  In IP6 2620:0:861:10d::11 > 2620:0:861:10d::253: ICMP6, echo request, seq 1, length 64
   cmooney@lsw1-f1-eqiad> monitor traffic interface irb.1035 matching "icmp6" no-resolve    
   verbose output suppressed, use <detail> or <extensive> for full protocol decode
   Address resolution is OFF.
   Listening on irb.1035, capture size 96 bytes
   
   12:40:49.343230  In IP6 2620:0:861:10d::11 > 2620:0:861:10d::1: ICMP6, echo request, seq 1, length 64
   12:40:49.433829 Out [|ip6]
   12:40:58.669189  In IP6 2620:0:861:10d::11 > 2620:0:861:10d::254: ICMP6, echo request, seq 1, length 64
   12:40:58.669212 Out [|ip6]

IPv6 Neighbors are as expected on both hosts:

   root@an-worker1147-test:~# ip -6 neigh show dev eno2np1.1035
   2620:0:861:10d::1 lladdr 00:00:5e:11:fa:ce router STALE
   2620:0:861:10d::253 lladdr a4:e1:1a:81:3a:80 router STALE
   2620:0:861:10d::254 lladdr a4:e1:1a:81:9e:80 router STALE
   root@an-worker1148-test:~# ip -6 neigh show 
   2620:0:861:10d::1 dev eno2np1 lladdr 00:00:5e:11:fa:ce router STALE
   2620:0:861:10d::253 dev eno2np1 lladdr a4:e1:1a:81:3a:80 router STALE
   2620:0:861:10d::254 dev eno2np1 lladdr a4:e1:1a:81:9e:80 router STALE

Inter-Vlan/subnet routing via IRB interfaces on same switch

Tests carried out on lsw1-e1-eqiad, between the IRB interfaces assocatied with private1-e1-eqiad and analytics-e1-eqiad (no filter in place for test).

Relevant Configuration


Relevant Config'

private1-e1-eqiad / ms-fe1012-test:

   cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1031 
   description private1-e1-eqiad;
   family inet {
       address 10.64.130.1/24;
   }
   family inet6 {
       address 2620:0:861:109::1/64;
   }
   root@ms-fe1012-test:~# ip -br addr show dev enp101s0f0np0
   enp101s0f0np0    UP             10.64.130.10/24 2620:0:861:109::10/64 fe80::e63d:1aff:fe12:b100/64 
   root@ms-fe1012-test:~# ip route show | grep default
   default via 10.64.130.1 dev enp101s0f0np0 
   root@ms-fe1012-test:~# ip -6 route show | grep default
   default via 2620:0:861:109::1 dev enp101s0f0np0 metric 1024 pref medium


analytics1-e1-eqiad / an-worker1148-test

   cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1039 
   description analytics1-e1-eqiad;
   family inet {
       address 10.64.138.1/24;
   }
   family inet6 {
       address 2620:0:861:100::1/64;
   }
   root@an-worker1147-test:/etc# ip -br addr show dev eno2np1
   eno2np1          UP             10.64.138.11/24 2620:0:861:100::11/64 fe80::e63d:1aff:fe54:1445/64 
   root@an-worker1147-test:/etc# ip route show | grep default
   default via 10.64.138.1 dev eno2np1 
   root@an-worker1147-test:/etc# ip -6 route show | grep default
   default via 2620:0:861:100::1 dev eno2np1 metric 1024 pref medium

IPv4

Results

   root@ms-fe1012-test:~# mtr --address 10.64.130.10 -b -w -c 5 10.64.138.11
   Start: 2022-02-10T17:07:22+0000
   HOST: ms-fe1012-test                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
     1.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (10.64.130.1)  0.0%     5    6.2   6.0   2.3   8.2   2.2
     2.|-- 10.64.138.11                                      0.0%     5    0.3   0.2   0.2   0.3   0.0
   root@an-worker1147-test:~# mtr --address 10.64.138.11 -b -w -c 5 10.64.130.10
   Start: 2022-02-10T17:06:33+0000
   HOST: an-worker1147-test Loss%   Snt   Last   Avg  Best  Wrst StDev
     1.|-- irb-1039.lsw1-e1-eqiad.eqiad.wmnet (10.64.138.1)  0.0%     5    5.7   4.5   0.8   9.9   3.6
     2.|-- 10.64.130.10        0.0%     5    0.2   0.2   0.2   0.2   0.0


IPv6

Results

   root@ms-fe1012-test:~# mtr --address 2620:0:861:109::10 -b -w -c 5 2620:0:861:100::11
   Start: 2022-02-10T17:08:11+0000
   HOST: ms-fe1012-test                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
     1.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:109::1)  0.0%     5    0.5   0.5   0.5   0.6   0.0
     2.|-- 2620:0:861:100::11                                      0.0%     5    0.2   0.2   0.2   0.2   0.0
   root@an-worker1147-test:~# mtr --address 2620:0:861:100::11 -b -w -c 5 2620:0:861:109::10
   Start: 2022-02-10T17:08:21+0000
   HOST: an-worker1147-test                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
     1.|-- irb-1039.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:100::1)  0.0%     5    0.6   0.5   0.5   0.6   0.0
     2.|-- 2620:0:861:109::10                                      0.0%     5    0.2   0.2   0.2   0.2   0.0

Inter-Vlan/subnet routing via IRB interfaces on separate switches

Tests carried out between ms-fe1012-test, connected to private1-e1-eqiad on lsw1-e1-eqiad, and an-worker1148-test, connected to analytics1-f1-eqiad on lsw1-f1-eqiad.

Relevant Config

private1-e1-eqiad / ms-fe1012-test:

   cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1031 
   description private1-e1-eqiad;
   family inet {
       address 10.64.130.1/24;
   }
   family inet6 {
       address 2620:0:861:109::1/64;
   }
   root@ms-fe1012-test:~# ip -br addr show dev enp101s0f0np0
   enp101s0f0np0    UP             10.64.130.10/24 2620:0:861:109::10/64 fe80::e63d:1aff:fe12:b100/64 
   root@ms-fe1012-test:~# ip route show | grep default
   default via 10.64.130.1 dev enp101s0f0np0 
   root@ms-fe1012-test:~# ip -6 route show | grep default
   default via 2620:0:861:109::1 dev enp101s0f0np0 metric 1024 pref medium


analytics1-f1-eqiad / an-worker1148-test:

   cmooney@lsw1-f1-eqiad> show configuration interfaces irb.1043  
   description analytics1-f1-eqiad;
   family inet {
       address 10.64.142.1/24;
   }
   family inet6 {
       address 2620:0:861:114::1/64;
   }
   root@an-worker1148-test:~# ip -br addr show dev eno2np1.1043
   eno2np1.1043@eno2np1 UP             10.64.142.10/24 2620:0:861:114::10/64 fe80::e63d:1aff:fe54:aba7/64 
   root@an-worker1148-test:~# ip route show | grep 10.0.0.0
   10.0.0.0/8 via 10.64.142.1 dev eno2np1.1043 
   root@an-worker1148-test:~# ip -6 route show | grep default
   default via 2620:0:861:114::1 dev eno2np1.1043 metric 1024 pref medium

IPv4

Results

   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 10.64.142.0 detail 
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
   10.64.142.0/24 (1 entry, 1 announced)
           *EVPN   Preference: 170/-101
                   Next hop type: Indirect, Next hop index: 0
                   Address: 0xc686b6c
                   Next-hop reference count: 15
                   Next hop type: Router, Next hop index: 1765
                   Next hop: 10.64.129.7 via et-0/0/52.0, selected
                   Session Id: 0x0
                   Protocol next hop: 10.64.128.7
                   Composite next hop: 0xbd31530 1760 INH Session ID: 0x0
                     VXLAN tunnel rewrite:
                       MTU: 0, Flags: 0x0
                       Encap table ID: 0, Decap table ID: 10
                       Encap VNI: 3005000, Decap VNI: 3005000
                       Source VTEP: 10.64.128.3, Destination VTEP: 10.64.128.7
                       SMAC: a4:e1:1a:81:3a:80, DMAC: a4:e1:1a:81:9e:80
                   Indirect next hop: 0xc816e84 524299 INH Session ID: 0x0
                   State: <Active Int Ext>
                   Age: 2d 0:43:19 	Metric2: 8 
                   Validation State: unverified 
                   Task: PRODUCTION-EVPN-L3-context
                   Announcement bits (2): 2-KRT 4-BGP_RT_Background 
                   AS path: I 
                   Thread: junos-main 
   cmooney@lsw1-f1-eqiad> show route table PRODUCTION.inet.0 10.64.130.0 detail 
   
   PRODUCTION.inet.0: 34 destinations, 37 routes (34 active, 0 holddown, 0 hidden)
   10.64.130.0/24 (1 entry, 1 announced)
           *EVPN   Preference: 170/-101
                   Next hop type: Indirect, Next hop index: 0
                   Address: 0xc68733c
                   Next-hop reference count: 21
                   Next hop type: Router, Next hop index: 1748
                   Next hop: 10.64.129.6 via et-0/0/52.0, selected
                   Session Id: 0x0
                   Protocol next hop: 10.64.128.3
                   Composite next hop: 0xbd309a0 1751 INH Session ID: 0x0
                     VXLAN tunnel rewrite:
                       MTU: 0, Flags: 0x0
                       Encap table ID: 0, Decap table ID: 8
                       Encap VNI: 3005000, Decap VNI: 3005000
                       Source VTEP: 10.64.128.7, Destination VTEP: 10.64.128.3
                       SMAC: a4:e1:1a:81:9e:80, DMAC: a4:e1:1a:81:3a:80
                   Indirect next hop: 0xc815b04 524294 INH Session ID: 0x0
                   State: <Active Int Ext>
                   Age: 6d 22:20:29 	Metric2: 8 
                   Validation State: unverified 
                   Task: PRODUCTION-EVPN-L3-context
                   Announcement bits (2): 2-KRT 3-BGP_RT_Background 
                   AS path: I 
                   Thread: junos-main 
   root@ms-fe1012-test:~# mtr --address 10.64.130.10 -c 5 -b -w 10.64.142.10
   Start: 2022-02-10T17:18:41+0000
   HOST: ms-fe1012-test                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
     1.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (10.64.130.1)  0.0%     5    3.1   4.7   1.4   7.6   2.6
     2.|-- irb-1043.lsw1-f1-eqiad.eqiad.wmnet (10.64.142.1)  0.0%     5    2.2   3.6   0.9   6.9   2.3
     3.|-- 10.64.142.10                                      0.0%     5    0.2   0.2   0.2   0.2   0.0
   root@an-worker1148-test:~# mtr --address 10.64.142.10 -c 5 -b -w 10.64.130.10 
   Start: 2022-02-10T16:58:30+0000
   HOST: an-worker1148-test                               Loss%   Snt   Last   Avg  Best  Wrst StDev
     1.|-- irb-1043.lsw1-f1-eqiad.eqiad.wmnet (10.64.142.1)  0.0%     5    3.7   6.8   3.7   8.4   1.9
     2.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (10.64.130.1)  0.0%     5    7.2   4.5   1.2   7.2   2.5
     3.|-- 10.64.130.10                                      0.0%     5    0.2   0.2   0.2   0.3   0.0

IPv6

Results

   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 2620:0:861:114:: detail 
   
   PRODUCTION.inet6.0: 40 destinations, 44 routes (40 active, 0 holddown, 0 hidden)
   2620:0:861:114::/64 (1 entry, 1 announced)
           *EVPN   Preference: 170/-101
                   Next hop type: Indirect, Next hop index: 0
                   Address: 0xc6862d4
                   Next-hop reference count: 13
                   Next hop type: Router, Next hop index: 1765
                   Next hop: 10.64.129.7 via et-0/0/52.0, selected
                   Session Id: 0x0
                   Protocol next hop: 10.64.128.7
                   Composite next hop: 0xbd314e0 1759 INH Session ID: 0x0
                     VXLAN tunnel rewrite:
                       MTU: 0, Flags: 0x0
                       Encap table ID: 0, Decap table ID: 10
                       Encap VNI: 3005000, Decap VNI: 3005000
                       Source VTEP: 10.64.128.3, Destination VTEP: 10.64.128.7
                       SMAC: a4:e1:1a:81:3a:80, DMAC: a4:e1:1a:81:9e:80
                   Indirect next hop: 0xc816d04 524298 INH Session ID: 0x0
                   State: <Active Int Ext>
                   Age: 2d 0:46:31 	Metric2: 8 
                   Validation State: unverified 
                   Task: PRODUCTION-EVPN-L3-context
                   Announcement bits (2): 2-KRT 4-BGP_RT_Background 
                   AS path: I 
                   Thread: junos-main 
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 2620:0:861:114:: detail 
   
   PRODUCTION.inet6.0: 37 destinations, 41 routes (37 active, 0 holddown, 0 hidden)
   2620:0:861:109::/64 (1 entry, 1 announced)
           *EVPN   Preference: 170/-101
                   Next hop type: Indirect, Next hop index: 0
                   Address: 0xc6865f4
                   Next-hop reference count: 19
                   Next hop type: Router, Next hop index: 1748
                   Next hop: 10.64.129.6 via et-0/0/52.0, selected
                   Session Id: 0x0
                   Protocol next hop: 10.64.128.3
                   Composite next hop: 0xbd30950 1750 INH Session ID: 0x0
                     VXLAN tunnel rewrite:
                       MTU: 0, Flags: 0x0
                       Encap table ID: 0, Decap table ID: 8
                       Encap VNI: 3005000, Decap VNI: 3005000
                       Source VTEP: 10.64.128.7, Destination VTEP: 10.64.128.3
                       SMAC: a4:e1:1a:81:9e:80, DMAC: a4:e1:1a:81:3a:80
                   Indirect next hop: 0xc815984 524293 INH Session ID: 0x0
                   State: <Active Int Ext>
                   Age: 6d 22:23:45 	Metric2: 8 
                   Validation State: unverified 
                   Task: PRODUCTION-EVPN-L3-context
                   Announcement bits (2): 2-KRT 3-BGP_RT_Background 
                   AS path: I 
                   Thread: junos-main 
   root@ms-fe1012-test:~# mtr --address 2620:0:861:109::10 -b -w -c 5 2620:0:861:114::10
   Start: 2022-02-10T17:18:04+0000
   HOST: ms-fe1012-test                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
     1.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:109::1)  0.0%     5    0.5   0.5   0.5   0.5   0.0
     2.|-- irb-1043.lsw1-f1-eqiad.eqiad.wmnet (2620:0:861:114::1)  0.0%     5    0.5   0.5   0.4   0.6   0.1
     3.|-- 2620:0:861:114::10                                      0.0%     5    0.3   0.3   0.3   0.3   0.0
   root@an-worker1148-test:~# mtr --address 2620:0:861:114::10 -c 5 -b -w 2620:0:861:109::10
   Start: 2022-02-10T17:00:30+0000
   HOST: an-worker1148-test                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
     1.|-- irb-1043.lsw1-f1-eqiad.eqiad.wmnet (2620:0:861:114::1)  0.0%     5    0.7   9.0   0.5  42.6  18.8
     2.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:109::1)  0.0%     5    0.6   0.5   0.5   0.6   0.0
     3.|-- 2620:0:861:109::10                                      0.0%     5    0.2   0.2   0.2   0.3   0.0

BGP Peering on Vlan segment to end device

These tests verify that end hosts can successfully create a BGP peering to the switch IRB interface of their connected subnet. This first set of tests validates the normal scenario where there is a per-rack subnet, and thus an IRB interface with only a single unicast IP.

Tests were done with ms-fe1012-test, connected to lsw1-e1-eqiad on Vlan1031 / private1-e1-eqiad. A public IP addresses (208.80.154.226/32 and 2620:0:861:ed1a::3/128) were allocated from the LVS public range and the peering defined as a standard LVS one in automation.

Relevant Config

   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group PyBal 
   type external;
   hold-time 30;
   import LVS_IMPORT;
   family inet {
       unicast {
           prefix-limit {
               maximum 1000;
               teardown 20;
           }
       }
   }
   family inet6 {
       unicast {
           prefix-limit {
               maximum 1000;
               teardown 20;
           }
       }
   }
   export NONE;
   peer-as 64600;
   neighbor 10.64.130.10 {
       description ms-fe1012-test;
   }

IPv4 only

   cmooney@lsw1-e1-eqiad> show bgp neighbor 10.64.130.10           
   
   Peer: 10.64.130.10+41482 AS 64600 Local: 10.64.130.1+179 AS 64810
     Description: ms-fe1012-test
     Group: PyBal                 Routing-Instance: PRODUCTION
     Forwarding routing-instance: PRODUCTION  
     Type: External    State: Established    Flags: <Sync>
     Last State: OpenConfirm   Last Event: RecvKeepAlive
     Last Error: None
     Export: [ NONE ] Import: [ LVS_IMPORT ]
     Options: <Preference HoldTime AddressFamily PeerAS PrefixLimit LocalAS Refresh>
     Options: <GracefulShutdownRcv>
     Address families configured: inet-unicast inet6-unicast
     Holdtime: 30 Preference: 170
     Graceful Shutdown Receiver local-preference: 0
     Local AS: 64810 Local System AS: 64810
     Number of flaps: 5
     Last flap event: RecvNotify
     Error: 'Cease' Sent: 0 Recv: 5
     Peer ID: 208.80.154.226  Local ID: 10.64.130.1       Active Holdtime: 30
     Keepalive Interval: 10         Group index: 10   Peer index: 0    SNMP index: 28    
     I/O Session Thread: bgpio-0 State: Enabled
     BFD: disabled, down
     Local Interface: irb.1031                         
     NLRI for restart configured on peer: inet-unicast inet6-unicast
     NLRI advertised by peer: inet-unicast
     NLRI for this session: inet-unicast
     Peer supports Refresh capability (2)
     Stale routes from peer are kept for: 300
     Peer does not support Restarter functionality
     NLRI that restart is negotiated for: inet-unicast
     NLRI of received end-of-rib markers: inet-unicast
     NLRI of all end-of-rib markers sent: inet-unicast
     Peer does not support LLGR Restarter or Receiver functionality
     Peer supports 4 byte AS extension (peer-as 64600)
     NLRI's for which peer can receive multiple paths: inet-unicast
     Table PRODUCTION.inet.0 Bit: 90001
       RIB State: BGP restart is complete
       RIB State: VPN restart is complete
       Send state: in sync
       Active prefixes:              1
       Received prefixes:            1
       Accepted prefixes:            1
       Suppressed due to damping:    0
       Advertised prefixes:          0
     Last traffic (seconds): Received 2    Sent 2    Checked 252 
     Input messages:  Total 29	Updates 2	Refreshes 0 	Octets 668
     Output messages: Total 28	Updates 0	Refreshes 0 	Octets 536
     Output Queue[8]: 0            (PRODUCTION.inet.0, inet-unicast)
   cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.130.10 table PRODUCTION.inet.0 
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 208.80.154.226/32       10.64.130.10         0                  64600 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 10.64.130.1 count 100 rapid 208.80.154.226    
   PING 208.80.154.226 (208.80.154.226): 56 data bytes
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   --- 208.80.154.226 ping statistics ---
   100 packets transmitted, 100 packets received, 0% packet loss
   round-trip min/avg/max/stddev = 0.298/7.792/26.546/2.866 ms


IPv4 carrying IPv4 & IPv6 address families

   cmooney@lsw1-e1-eqiad> show bgp neighbor 10.64.130.10                                                                                  
   
   Peer: 10.64.130.10+41484 AS 64600 Local: 10.64.130.1+179 AS 64810
     Description: ms-fe1012-test
     Group: PyBal                 Routing-Instance: PRODUCTION
     Forwarding routing-instance: PRODUCTION  
     Type: External    State: Established    Flags: <Sync>
     Last State: OpenConfirm   Last Event: RecvKeepAlive
     Last Error: None
     Export: [ NONE ] Import: [ LVS_IMPORT ]
     Options: <Preference HoldTime AddressFamily PeerAS PrefixLimit LocalAS Refresh>
     Options: <GracefulShutdownRcv>
     Address families configured: inet-unicast inet6-unicast
     Holdtime: 30 Preference: 170
     Graceful Shutdown Receiver local-preference: 0
     Local AS: 64810 Local System AS: 64810
     Number of flaps: 6
     Last flap event: RecvNotify
     Error: 'Cease' Sent: 0 Recv: 6
     Peer ID: 208.80.154.226  Local ID: 10.64.130.1       Active Holdtime: 30
     Keepalive Interval: 10         Group index: 10   Peer index: 0    SNMP index: 28    
     I/O Session Thread: bgpio-0 State: Enabled
     BFD: disabled, down
     Local Interface: irb.1031                         
     NLRI for restart configured on peer: inet-unicast inet6-unicast
     NLRI advertised by peer: inet-unicast inet6-unicast
     NLRI for this session: inet-unicast inet6-unicast
     Peer supports Refresh capability (2)
     Stale routes from peer are kept for: 300
     Peer does not support Restarter functionality
     NLRI that restart is negotiated for: inet-unicast inet6-unicast
     NLRI of received end-of-rib markers: inet-unicast inet6-unicast
     NLRI of all end-of-rib markers sent: inet-unicast inet6-unicast
     Peer does not support LLGR Restarter or Receiver functionality
     Peer supports 4 byte AS extension (peer-as 64600)
     NLRI's for which peer can receive multiple paths: inet-unicast inet6-unicast
     Table PRODUCTION.inet.0 Bit: 90001
       RIB State: BGP restart is complete
       RIB State: VPN restart is complete
       Send state: in sync
       Active prefixes:              1
       Received prefixes:            1
       Accepted prefixes:            1
       Suppressed due to damping:    0
       Advertised prefixes:          0
     Table PRODUCTION.inet6.0 Bit: a0001
       RIB State: BGP restart is complete
       RIB State: VPN restart is complete
       Send state: in sync
       Active prefixes:              0
       Received prefixes:            0
       Accepted prefixes:            0
       Suppressed due to damping:    0     
       Advertised prefixes:          0
     Last traffic (seconds): Received 3    Sent 4    Checked 13  
     Input messages:  Total 6	Updates 3 	Refreshes 0 	Octets 253
     Output messages: Total 4	Updates 0 	Refreshes 0 	Octets 91
     Output Queue[8]: 0            (PRODUCTION.inet.0, inet-unicast)
     Output Queue[9]: 0            (PRODUCTION.inet6.0, inet6-unicast)


   cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.130.10 table PRODUCTION.inet.0            
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 208.80.154.226/32       10.64.130.10         0                  64600 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 10.64.130.1 count 100 rapid 208.80.154.226    
   PING 208.80.154.226 (208.80.154.226): 56 data bytes
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


   cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.130.10 table PRODUCTION.inet6.0           
   
   PRODUCTION.inet6.0: 40 destinations, 44 routes (40 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
     2620:0:861:ed1a::3/128
   *                         2620:0:861:109::10   0                  64600 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 2620:0:861:109::1 count 100 rapid 2620:0:861:ed1a::3 
   PING6(56=40+8+8 bytes) 2620:0:861:109::1 --> 2620:0:861:ed1a::3
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


IPv4 & IPv6 each carrying their own address family

These tests were instead carried out from an-worker1147-test to lsw1-e1-eqiad. Two BGP peerings are defined towards the host, one in the 'Kubernetes4' group and the other in 'Kubernetes6'. Peering was within Vlan1039 / analytics1-e1-eqiad for both address types.

Relevant Config

   cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1039 
   description analytics1-e1-eqiad;
   family inet {
       address 10.64.138.1/24;
   }
   family inet6 {
       address 2620:0:861:100::1/64;
   }
   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group Kubernetes4
   type external;
   multihop {
       ttl 2;
   }
   hold-time 30;
   import kubernetes_import;
   family inet {
       unicast {
           prefix-limit {
               maximum 2000;
               teardown 80;
           }
       }
   }
   export NONE;
   peer-as 64601;
   multipath;
   neighbor 10.64.138.11 {
       description an-worker1147-test;
   }
   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group Kubernetes6   
   type external;
   multihop {
       ttl 2;
   }
   hold-time 30;
   import kubernetes_import;
   family inet6 {
       unicast {
           prefix-limit {
               maximum 2000;
               teardown 80;
           }
       }
   }
   export NONE;
   peer-as 64601;
   multipath;
   neighbor 2620:0:861:100::11 {
       description an-worker1147-test;
   }

Results

   cmooney@lsw1-e1-eqiad> show bgp neighbor 10.64.138.11  
   
   Peer: 10.64.138.11+60646 AS 64601 Local: 10.64.138.1+179 AS 64810
     Description: an-worker1147-test
     Group: Kubernetes4           Routing-Instance: PRODUCTION
     Forwarding routing-instance: PRODUCTION  
     Type: External    State: Established    Flags: <Sync>
     Last State: OpenConfirm   Last Event: Refresh
     Last Error: None
     Export: [ NONE ] Import: [ kubernetes_import ]
     Options: <Multihop Preference HoldTime Ttl AddressFamily PeerAS Multipath PrefixLimit LocalAS Refresh>
     Options: <GracefulShutdownRcv>
     Address families configured: inet-unicast
     Holdtime: 30 Preference: 170
     Graceful Shutdown Receiver local-preference: 0
     Local AS: 64810 Local System AS: 64810
     Number of flaps: 0
     Peer ID: 208.80.154.228  Local ID: 10.64.130.1       Active Holdtime: 30
     Keepalive Interval: 10         Group index: 12   Peer index: 0    SNMP index: 29    
     I/O Session Thread: bgpio-0 State: Enabled
     BFD: disabled, down
     NLRI for restart configured on peer: inet-unicast
     NLRI advertised by peer: inet-unicast
     NLRI for this session: inet-unicast
     Peer supports Refresh capability (2)
     Stale routes from peer are kept for: 300
     Peer does not support Restarter functionality
     NLRI that restart is negotiated for: inet-unicast
     NLRI of received end-of-rib markers: inet-unicast
     NLRI of all end-of-rib markers sent: inet-unicast
     Peer does not support LLGR Restarter or Receiver functionality
     Peer supports 4 byte AS extension (peer-as 64601)
     NLRI's for which peer can receive multiple paths: inet-unicast
     Table PRODUCTION.inet.0 Bit: 90002
       RIB State: BGP restart is complete
       RIB State: VPN restart is complete
       Send state: in sync
       Active prefixes:              0
       Received prefixes:            1
       Accepted prefixes:            0
       Suppressed due to damping:    0
       Advertised prefixes:          0
     Last traffic (seconds): Received 5    Sent 6    Checked 615 
     Input messages:  Total 69	Updates 5	Refreshes 1 	Octets 1520
     Output messages: Total 69	Updates 0	Refreshes 0 	Octets 1315
     Output Queue[8]: 0            (PRODUCTION.inet.0, inet-unicast)
   cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.138.11 table PRODUCTION.inet.0                  
   
   PRODUCTION.inet.0: 35 destinations, 39 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
     208.80.154.228/32       10.64.138.11         0                  64601 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 10.64.138.1 count 100 rapid 208.80.154.228     
   PING 208.80.154.228 (208.80.154.228): 56 data bytes
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


   cmooney@lsw1-e1-eqiad> show bgp neighbor 2620:0:861:100::11 
   
   Peer: 2620:0:861:100::11+41120 AS 64601 Local: 2620:0:861:100::1+179 AS 64810
     Description: an-worker1147-test
     Group: Kubernetes6           Routing-Instance: PRODUCTION
     Forwarding routing-instance: PRODUCTION  
     Type: External    State: Established    Flags: <Sync>
     Last State: OpenConfirm   Last Event: Refresh
     Last Error: Open Message Error
     Export: [ NONE ] Import: [ kubernetes_import ]
     Options: <Multihop Preference HoldTime Ttl AddressFamily PeerAS Multipath PrefixLimit LocalAS Refresh>
     Options: <GracefulShutdownRcv>
     Address families configured: inet6-unicast
     Holdtime: 30 Preference: 170
     Graceful Shutdown Receiver local-preference: 0
     Local AS: 64810 Local System AS: 64810
     Number of flaps: 0
     Error: 'Open Message Error' Sent: 4 Recv: 0
     Peer ID: 208.80.154.228  Local ID: 10.64.130.1       Active Holdtime: 30
     Keepalive Interval: 10         Group index: 13   Peer index: 0    SNMP index: 30    
     I/O Session Thread: bgpio-0 State: Enabled
     BFD: disabled, down
     NLRI for restart configured on peer: inet6-unicast
     NLRI advertised by peer: inet6-unicast
     NLRI for this session: inet6-unicast
     Peer supports Refresh capability (2)
     Stale routes from peer are kept for: 300
     Peer does not support Restarter functionality
     NLRI that restart is negotiated for: inet6-unicast
     NLRI of received end-of-rib markers: inet6-unicast
     NLRI of all end-of-rib markers sent: inet6-unicast
     Peer does not support LLGR Restarter or Receiver functionality
     Peer supports 4 byte AS extension (peer-as 64601)
     NLRI's for which peer can receive multiple paths: inet6-unicast
     Table PRODUCTION.inet6.0 Bit: a0002
       RIB State: BGP restart is complete
       RIB State: VPN restart is complete
       Send state: in sync
       Active prefixes:              0
       Received prefixes:            1
       Accepted prefixes:            0
       Suppressed due to damping:    0
       Advertised prefixes:          0
     Last traffic (seconds): Received 8    Sent 1    Checked 348 
     Input messages:  Total 43	Updates 6	Refreshes 1 	Octets 1256
     Output messages: Total 39	Updates 0	Refreshes 0 	Octets 752
     Output Queue[9]: 0            (PRODUCTION.inet6.0, inet6-unicast)
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:100::11                                                                                                               
   
   PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
     2620:0:861:ed1a::4/128
   *                         2620:0:861:100::11   0                  64601 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 2620:0:861:100::1 count 100 rapid 2620:0:861:ed1a::4   
   PING6(56=40+8+8 bytes) 2620:0:861:100::1 --> 2620:0:861:ed1a::4
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

BGP Peering on Vlan segment from Anycast GW IP

BGP peering for these tests was carried out from an-worker1147-test, connected to lsw1-e1-eqiad. Vlan private1-f1-eqiad was extended to this switch, and an Anycast GW configuration added to the IRB interface to support this.

Relevant Config

   cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1035    
   virtual-gateway-accept-data;
   description private1-f1-eqiad;
   family inet {
       address 10.64.134.253/24 {
           preferred;
           virtual-gateway-address 10.64.134.1;
       }
   }
   family inet6 {
       address 2620:0:861:10d::253/64 {
           preferred;
           virtual-gateway-address 2620:0:861:10d::1;
       }
   }
   virtual-gateway-v4-mac 00:00:5e:11:fa:ce;
   virtual-gateway-v6-mac 00:00:5e:11:fa:ce;


IPv4 only

Peering was done by adding the IP address of an-worker1147-test on Vlan1135 to the "Anycast4" BGP group:

   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group Anycast4 
   type external;
   multihop {
       ttl 193;
   }
   damping;
   import ANYCAST_IMPORT;
   family inet {
       unicast {
           prefix-limit {
               maximum 50;
               teardown 80;
           }
       }
   }
   export NONE;
   peer-as 64605;
   multipath;
   bfd-liveness-detection {
       minimum-interval 300;
   }
   neighbor 10.64.134.72 {
       description an-worker1147-test;
   }

Connection was established from the an-worker1147-test IP on Vlan1135, to Anycast IP 10.64.134.1 (using FRR BGP stack), public IP was announced:

   an-worker1147-test# show bgp ipv4 unicast neighbors 10.64.134.1 advertised-routes 
   BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
   Default local pref 100, local AS 64605
   Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                  i internal, r RIB-failure, S Stale, R Removed
   Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
   Origin codes:  i - IGP, e - EGP, ? - incomplete
   RPKI validation codes: V valid, I invalid, N Not found
   
      Network          Next Hop            Metric LocPrf Weight Path
   *> 208.80.154.228/32
                       0.0.0.0                  0         32768 ?


Route is accepted by the top of rack and public IP is pingable:

   cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet.0                  
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 208.80.154.228/32       10.64.134.72         0                  64605 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION rapid count 100 source 10.64.134.1 208.80.154.228
   PING 208.80.154.228 (208.80.154.228): 56 data bytes
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
   Start: 2022-02-14T17:18:04+0000
   HOST: nbgw                                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS6830   176.61.34.1                                                       0.0%    10   14.5  15.0  11.9  20.5   2.8
     2. AS6830   109.255.255.254                                                   0.0%    10   11.3  16.4   7.6  55.0  13.8
     3. AS6830   ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42)                   0.0%    10   10.1  15.9  10.1  30.4   6.0
     4. AS6830   ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110)                  0.0%    10   10.6  15.4   9.0  45.5  10.8
     5. AS1299   dln-b2-link.ip.twelve99.net (62.115.172.136)                      0.0%    10   19.1  13.3   7.7  20.4   4.2
     6. AS1299   ldn-bb1-link.ip.twelve99.net (62.115.120.100)                     0.0%    10   20.3  21.5  15.7  26.3   3.0
     7. AS1299   nyk-bb2-link.ip.twelve99.net (62.115.113.20)                      0.0%    10   94.7  97.3  94.4 100.2   2.4
     8. AS1299   ash-bb2-link.ip.twelve99.net (62.115.136.201)                     0.0%    10   99.2 101.2  97.1 105.5   3.1
     9. AS1299   ash-b1-link.ip.twelve99.net (62.115.143.121)                      0.0%    10  101.7 103.3 100.6 108.5   2.5
    10. AS1299   wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226)   0.0%    10  101.4  99.7  95.6 103.2   2.4
    11. AS14907  ae0.cr1-eqiad.wikimedia.org (208.80.154.193)                      0.0%    10  105.5 103.1  97.0 110.5   5.0
    12. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
    13. AS14907  208.80.154.228                                                    0.0%    10  101.4  99.4  94.8 104.6   3.0


IPv4 carrying IPv4 & IPv6 address families

For this test the IP of an-worker1147-test on Vlan1035 was placed into the 'PyBal' group in the overlay BGP config, which means the switch will announce the capability for IPv4 and IPv6 address families over the single IPv4 peering.

   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group PyBal 
   type external;
   hold-time 30;
   import LVS_IMPORT;
   family inet {
       unicast {
           prefix-limit {
               maximum 1000;
               teardown 20;
           }
       }
   }
   family inet6 {
       unicast {
           prefix-limit {
               maximum 1000;
               teardown 20;
           }
       }
   }
   export NONE;
   peer-as 64600;
   neighbor 10.64.134.72 {
       description an-worker1147-test;
   }


Host was configured to send both types of route over the session:

   an-worker1147-test# show bgp ipv4 neighbors 10.64.134.1 advertised-routes 
   BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
   Default local pref 100, local AS 64600
   Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                  i internal, r RIB-failure, S Stale, R Removed
   Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
   Origin codes:  i - IGP, e - EGP, ? - incomplete
   RPKI validation codes: V valid, I invalid, N Not found
   
      Network          Next Hop            Metric LocPrf Weight Path
   *> 208.80.154.228/32
                       0.0.0.0                  0         32768 ?
   an-worker1147-test# show bgp ipv6 neighbors 10.64.134.1 advertised-routes 
   BGP table version is 3, local router ID is 208.80.154.228, vrf id 0
   Default local pref 100, local AS 64600
   Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                  i internal, r RIB-failure, S Stale, R Removed
   Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
   Origin codes:  i - IGP, e - EGP, ? - incomplete
   RPKI validation codes: V valid, I invalid, N Not found
   
      Network          Next Hop            Metric LocPrf Weight Path
   *> 2620:0:861:ed1a::4/128
                       ::                       0         32768 ?


IPv4 received and works:

   cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet.0    
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 208.80.154.228/32       10.64.134.72         0                  64600 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 208.80.154.228 rapid count 100 source 10.64.134.253              
   PING 208.80.154.228 (208.80.154.228): 56 data bytes
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
   Start: 2022-02-14T17:42:02+0000
   HOST: nbgw                                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS6830   176.61.34.1                                                       0.0%    10   13.5  16.0  10.3  26.0   4.8
     2. AS6830   109.255.255.254                                                   0.0%    10   17.2  13.3   9.8  17.2   2.4
     3. AS6830   ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42)                   0.0%    10   11.6  12.3   8.9  17.1   2.9
     4. AS6830   ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110)                  0.0%    10   28.3  13.4   9.0  28.3   5.7
     5. AS1299   dln-b2-link.ip.twelve99.net (62.115.172.136)                      0.0%    10   13.2  11.8   9.1  14.3   1.8
     6. AS1299   ldn-bb1-link.ip.twelve99.net (62.115.120.100)                     0.0%    10   22.4  23.3  19.0  33.0   3.7
     7. AS1299   nyk-bb2-link.ip.twelve99.net (62.115.113.20)                      0.0%    10   96.4  97.9  92.7 105.9   4.0
     8. AS1299   ash-bb2-link.ip.twelve99.net (62.115.136.201)                     0.0%    10  100.2 102.6  93.4 125.7   9.2
     9. AS1299   ash-b1-link.ip.twelve99.net (62.115.143.121)                      0.0%    10   98.5 102.6  97.1 106.7   3.1
    10. AS1299   wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226)   0.0%    10  100.0 101.5  98.3 105.5   2.8
    11. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
    12. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
    13. AS14907  208.80.154.228                                                    0.0%    10   98.0  98.5  94.9 106.3   3.5


IPv6 also received and works:

   cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet6.0   
   
   PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
     2620:0:861:ed1a::4/128
   *                         2620:0:861:10d::72   0                  64600 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 2620:0:861:ed1a::4 rapid count 100 source 2620:0:861:10d::253 
   PING6(56=40+8+8 bytes) 2620:0:861:10d::253 --> 2620:0:861:ed1a::4
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   cathal@nbgw:~$ mtr -b -w -z -c 10 2620:0:861:ed1a::4
   Start: 2022-02-14T17:39:50+0000
   HOST: nbgw                                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS6939   tunnel650354.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:32c::1)  10.0%    10   23.9  28.5  22.1  43.7   7.8
     2. AS6939   e0-19.core2.lon2.he.net (2001:470:0:67::1)                           0.0%    10   35.8  41.3  23.8 110.6  28.7
     3. AS6939   100ge11-2.core1.lon2.he.net (2001:470:0:541::1)                      0.0%    10   24.7  29.0  20.1  61.5  14.2
     4. AS6939   100ge4-1.core1.nyc4.he.net (2001:470:0:2cf::2)                       0.0%    10  103.7  88.7  81.4 103.7   6.1
     5. AS6939   100ge11-1.core1.nyc5.he.net (2001:470:0:20a::2)                      0.0%    10   90.4  93.2  84.8 138.5  16.2
     6. AS???    ???                                                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     7. AS6939   100ge1-2.core1.ash1.he.net (2001:470:0:277::1)                       0.0%    10  108.1 114.2  94.7 188.9  26.9
     8. AS6939   xe-5-3-3-500.cr1-eqiad.wikimedia.org (2001:470:0:1c0::2)             0.0%    10   96.3 101.0  90.3 130.5  14.0
     9. AS14907  et-0-0-48-100.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:fe07::2)        10.0%    10   94.6 101.7  92.6 155.8  20.3
    10. AS14907  2620:0:861:ed1a::4                                                   0.0%    10  110.5 103.3  89.6 140.9  16.9


IPv4 & IPv6 each carrying their own address family

For the final test separate BGP peering sessions to an-worker1147-test were established from lsw1-e1-eqiad, over both IPv4 and IPv6, exchanging just the routes for a single address family over each.

   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group Kubernetes4  
   type external;
   multihop {
       ttl 2;
   }
   hold-time 30;
   import kubernetes_import;
   family inet {
       unicast {
           prefix-limit {
               maximum 2000;
               teardown 80;
           }
       }
   }
   export NONE;
   peer-as 64601;
   multipath;
   neighbor 10.64.134.72 {
       description an-worker1147-test;
   }
   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group Kubernetes6    
   type external;
   multihop {
       ttl 2;
   }
   hold-time 30;
   import kubernetes_import;
   family inet6 {
       unicast {
           prefix-limit {
               maximum 2000;
               teardown 80;
           }
       }
   }
   export NONE;
   peer-as 64601;
   multipath;
   neighbor 2620:0:861:10d::72 {
       description an-worker1147-test;
   }

IPv4 results:

   an-worker1147-test# show bgp ipv4 neighbors 10.64.134.1 advertised-routes 
   BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
   Default local pref 100, local AS 64601
   Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                  i internal, r RIB-failure, S Stale, R Removed
   Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
   Origin codes:  i - IGP, e - EGP, ? - incomplete
   RPKI validation codes: V valid, I invalid, N Not found
   
      Network          Next Hop            Metric LocPrf Weight Path
   *> 208.80.154.228/32
                       0.0.0.0                  0         32768 ?
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.64.134.72 
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 208.80.154.228/32       10.64.134.72         0                  64601 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 208.80.154.228 rapid count 100 source 10.64.134.253    
   PING 208.80.154.228 (208.80.154.228): 56 data bytes
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
   Start: 2022-02-14T18:10:25+0000
   HOST: nbgw                                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS6830   176.61.34.1                                                       0.0%    10   18.3  15.4  10.1  20.6   3.8
     2. AS6830   109.255.255.254                                                   0.0%    10   16.0  18.2   8.5  50.2  12.2
     3. AS6830   ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42)                   0.0%    10   15.2  19.1  10.4  51.6  12.3
     4. AS6830   ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110)                  0.0%    10   12.9  17.8   8.5  53.9  13.0
     5. AS1299   dln-b2-link.ip.twelve99.net (62.115.172.136)                      0.0%    10   15.9  14.6   9.5  19.5   3.0
     6. AS1299   ldn-bb1-link.ip.twelve99.net (62.115.120.100)                     0.0%    10   23.3  28.3  20.1  53.7  10.0
     7. AS1299   nyk-bb2-link.ip.twelve99.net (62.115.113.20)                      0.0%    10  106.9  98.9  94.2 106.9   4.2
     8. AS1299   ash-bb2-link.ip.twelve99.net (62.115.136.201)                     0.0%    10  112.7 105.3  98.4 120.2   6.9
     9. AS1299   ash-b1-link.ip.twelve99.net (62.115.143.121)                      0.0%    10  104.5 106.9 101.0 125.8   7.6
    10. AS1299   wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226)   0.0%    10  111.7 106.2  96.1 112.7   6.0
    11. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
    12. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
    13. AS14907  208.80.154.228                                                    0.0%    10  104.1 106.2  93.8 125.3  11.0


IPv6 routes:

   an-worker1147-test# show bgp ipv6 neighbors 2620:0:861:10d::1 advertised-routes 
   BGP table version is 3, local router ID is 208.80.154.228, vrf id 0
   Default local pref 100, local AS 64601
   Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                  i internal, r RIB-failure, S Stale, R Removed
   Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
   Origin codes:  i - IGP, e - EGP, ? - incomplete
   RPKI validation codes: V valid, I invalid, N Not found
   
      Network          Next Hop            Metric LocPrf Weight Path
   *> 2620:0:861:ed1a::4/128
                       ::                       0         32768 ?
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:10d::72   
   
   PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
     2620:0:861:ed1a::4/128
   *                         2620:0:861:10d::72   0                  64601 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 2620:0:861:ed1a::4 rapid count 100 source 2620:0:861:10d::253    
   PING6(56=40+8+8 bytes) 2620:0:861:10d::253 --> 2620:0:861:ed1a::4
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   cathal@nbgw:~$ mtr -b -w -z -c 10 2620:0:861:ed1a::4
   Start: 2022-02-14T18:11:17+0000
   HOST: nbgw                                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS6939   tunnel650354.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:32c::1)  10.0%    10   25.2  30.0  22.3  41.5   7.8
     2. AS6939   e0-19.core2.lon2.he.net (2001:470:0:67::1)                          10.0%    10  124.9  43.1  23.1 124.9  32.7
     3. AS6939   100ge11-2.core1.lon2.he.net (2001:470:0:541::1)                      0.0%    10   36.2  31.4  20.5  59.4  11.4
     4. AS6939   100ge4-1.core1.nyc4.he.net (2001:470:0:2cf::2)                       0.0%    10   92.6  97.4  85.0 147.7  18.6
     5. AS6939   100ge11-1.core1.nyc5.he.net (2001:470:0:20a::2)                      0.0%    10   93.9  98.6  85.1 142.3  16.6
     6. AS???    ???                                                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     7. AS6939   100ge1-2.core1.ash1.he.net (2001:470:0:277::1)                       0.0%    10   94.1 109.7  93.2 193.9  30.8
     8. AS6939   xe-5-3-3-500.cr1-eqiad.wikimedia.org (2001:470:0:1c0::2)             0.0%    10  116.2 100.1  90.1 130.0  12.8
     9. AS14907  et-0-0-48-100.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:fe07::2)         0.0%    10  109.6 101.6  92.6 126.9  10.4
    10. AS14907  2620:0:861:ed1a::4                                                   0.0%    10   93.6 100.2  84.7 134.9  15.9
   

NOTE: This proves we can peer from the Anycast IP, but the switches will normally try to form the adjacency from their individual IP on the subnet. We can control this with "update source" on the switch if we ever need to, for the purpose of this test the only requirement was to confirm it will work if we need to do it.

BGP Peering on Vlan segment with Anycast GW, from Unicast IP

The purpose of this test is to validate that the QFX will allow a BGP adjacency to form to a device, connected via an access or trunk port, to its unique IRB interface IP on an int that has an anycast GW configured also.

Typically we want to peer directly with the GW. But if we ever run MC-LAG between two devices this would be problematic, two switches would be sharing this IP. Adjacency would form, but it would be random and only to a single device. So if we ever have that scenario we instead want to peer to each switch separately, using their unique IPs on that subnet rather than the shared VIP.

Relevant Config

The 3 scenarios are identical to in the previous set of tests (peering to the anycast VIP). Zero changes have been made on the QFX side, just the IP address configured as BGP peer has been adjusted on the test hosts in each case.

IPv4 Only

   an-worker1147-test# show bgp ipv4 unicast neighbors 10.64.134.253 advertised-routes 
   BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
   Default local pref 100, local AS 64605
   Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                  i internal, r RIB-failure, S Stale, R Removed
   Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
   Origin codes:  i - IGP, e - EGP, ? - incomplete
   RPKI validation codes: V valid, I invalid, N Not found
   
      Network          Next Hop            Metric LocPrf Weight Path
   *> 208.80.154.228/32
                       0.0.0.0                  0         32768 ?


Route is accepted by the top of rack and public IP is pingable:

   cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet.0                  
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 208.80.154.228/32       10.64.134.72         0                  64605 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION rapid count 100 source 10.64.134.1 208.80.154.228
   PING 208.80.154.228 (208.80.154.228): 56 data bytes
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
   Start: 2022-02-15T17:31:07+0000
   HOST: nbgw                                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS6830   176.61.34.1                                                       0.0%    10   14.2  15.0  11.2  20.5   2.8
     2. AS6830   109.255.255.254                                                   0.0%    10   11.1  16.4   7.6  55.0  13.8
     3. AS6830   ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42)                   0.0%    10   10.1  15.9  10.1  30.4   6.0
     4. AS6830   ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110)                  0.0%    10   10.6  15.4   9.0  45.5  10.8
     5. AS1299   dln-b2-link.ip.twelve99.net (62.115.172.136)                      0.0%    10   19.0  13.3   7.7  20.4   4.2
     6. AS1299   ldn-bb1-link.ip.twelve99.net (62.115.120.100)                     0.0%    10   20.3  21.5  15.7  26.3   3.0
     7. AS1299   nyk-bb2-link.ip.twelve99.net (62.115.113.20)                      0.0%    10   94.1  97.3  94.4 100.2   2.4
     8. AS1299   ash-bb2-link.ip.twelve99.net (62.115.136.201)                     0.0%    10   99.2 101.2  97.1 105.5   3.1
     9. AS1299   ash-b1-link.ip.twelve99.net (62.115.143.121)                      0.0%    10  101.7 103.3 100.6 108.5   2.5
    10. AS1299   wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226)   0.0%    10  101.4  99.7  95.6 103.2   2.4
    11. AS14907  ae0.cr1-eqiad.wikimedia.org (208.80.154.193)                      0.0%    10  105.5 103.1  97.0 110.5   5.0
    12. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
    13. AS14907  208.80.154.228                                                    0.0%    10  101.4  99.4  94.8 104.6   3.0


IPv4 carrying IPv4 & IPv6 address families

Host was configured to send both types of route over the session:

   an-worker1147-test# show bgp ipv4 neighbors 10.64.134.253 advertised-routes 
   BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
   Default local pref 100, local AS 64600
   Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                  i internal, r RIB-failure, S Stale, R Removed
   Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
   Origin codes:  i - IGP, e - EGP, ? - incomplete
   RPKI validation codes: V valid, I invalid, N Not found
   
      Network          Next Hop            Metric LocPrf Weight Path
   *> 208.80.154.228/32
                       0.0.0.0                  0         32768 ?
   an-worker1147-test# show bgp ipv6 neighbors 10.64.134.253 advertised-routes 
   BGP table version is 3, local router ID is 208.80.154.228, vrf id 0
   Default local pref 100, local AS 64600
   Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                  i internal, r RIB-failure, S Stale, R Removed
   Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
   Origin codes:  i - IGP, e - EGP, ? - incomplete
   RPKI validation codes: V valid, I invalid, N Not found
   
      Network          Next Hop            Metric LocPrf Weight Path
   *> 2620:0:861:ed1a::4/128
                       ::                       0         32768 ?


IPv4 received and works:

   cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet.0    
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 208.80.154.228/32       10.64.134.72         0                  64600 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 208.80.154.228 rapid count 100 source 10.64.134.253              
   PING 208.80.154.228 (208.80.154.228): 56 data bytes
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
   Start: 2022-02-14T18:05:01+0000
   HOST: nbgw                                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS6830   176.61.34.1                                                       0.0%    10   12.5  15.0  10.3  26.0   4.8
     2. AS6830   109.255.255.254                                                   0.0%    10   17.2  13.3   9.8  17.2   2.4
     3. AS6830   ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42)                   0.0%    10   11.6  12.3   8.9  17.1   2.9
     4. AS6830   ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110)                  0.0%    10   28.3  13.4   9.0  28.3   5.7
     5. AS1299   dln-b2-link.ip.twelve99.net (62.115.172.136)                      0.0%    10   13.2  11.8   9.1  14.3   1.8
     6. AS1299   ldn-bb1-link.ip.twelve99.net (62.115.120.100)                     0.0%    10   22.4  23.3  19.0  33.0   3.7
     7. AS1299   nyk-bb2-link.ip.twelve99.net (62.115.113.20)                      0.0%    10   96.4  97.8  92.7 105.9   4.0
     8. AS1299   ash-bb2-link.ip.twelve99.net (62.115.136.201)                     0.0%    10  100.2 102.6  93.4 125.7   9.2
     9. AS1299   ash-b1-link.ip.twelve99.net (62.115.143.121)                      0.0%    10   98.5 102.6  97.1 106.7   3.1
    10. AS1299   wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226)   0.0%    10  100.0 101.5  98.3 105.5   2.8
    11. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
    12. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
    13. AS14907  208.80.154.228                                                    0.0%    10   98.0  98.5  94.9 106.3   3.5


IPv6 also received and works:

   cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet6.0   
   
   PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
     2620:0:861:ed1a::4/128
   *                         2620:0:861:10d::72   0                  64600 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 2620:0:861:ed1a::4 rapid count 100 source 2620:0:861:10d::253 
   PING6(56=40+8+8 bytes) 2620:0:861:10d::253 --> 2620:0:861:ed1a::4
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   cathal@nbgw:~$ mtr -b -w -z -c 10 2620:0:861:ed1a::4
   Start: 2022-02-14T17:39:50+0000
   HOST: nbgw                                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS6939   tunnel650354.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:32c::1)  10.0%    10   23.9  28.5  22.1  43.7   7.8
     2. AS6939   e0-19.core2.lon2.he.net (2001:470:0:67::1)                           0.0%    10   35.8  41.3  23.8 110.6  28.7
     3. AS6939   100ge11-2.core1.lon2.he.net (2001:470:0:541::1)                      0.0%    10   24.7  29.0  20.1  61.5  14.2
     4. AS6939   100ge4-1.core1.nyc4.he.net (2001:470:0:2cf::2)                       0.0%    10  103.7  88.7  81.4 103.7   6.1
     5. AS6939   100ge11-1.core1.nyc5.he.net (2001:470:0:20a::2)                      0.0%    10   90.4  93.2  84.8 138.5  16.2
     6. AS???    ???                                                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     7. AS6939   100ge1-2.core1.ash1.he.net (2001:470:0:277::1)                       0.0%    10  108.1 114.2  94.7 188.9  26.9
     8. AS6939   xe-5-3-3-500.cr1-eqiad.wikimedia.org (2001:470:0:1c0::2)             0.0%    10   96.3 101.0  90.3 130.5  14.0
     9. AS14907  et-0-0-48-100.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:fe07::2)        10.0%    10   94.6 101.7  92.6 155.8  20.3
    10. AS14907  2620:0:861:ed1a::4                                                   0.0%    10  110.5 103.3  89.6 140.9  16.9


IPv4 & IPv6 each carrying their own address family

IPv4 session up and route working:

   an-worker1147-test# show bgp ipv4 neighbors 10.64.134.253 advertised-routes 
   BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
   Default local pref 100, local AS 64601
   Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                  i internal, r RIB-failure, S Stale, R Removed
   Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
   Origin codes:  i - IGP, e - EGP, ? - incomplete
   RPKI validation codes: V valid, I invalid, N Not found
   
      Network          Next Hop            Metric LocPrf Weight Path
   *> 208.80.154.228/32
                       0.0.0.0                  0         32768 ?
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.64.134.72                                                                                                                      
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 208.80.154.228/32       10.64.134.72         0                  64601 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 10.64.134.1 rapid count 100 208.80.154.228     
   PING 208.80.154.228 (208.80.154.228): 56 data bytes
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
   Start: 2022-02-15T17:19:44+0000
   HOST: nbgw                                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS6830   176.61.34.1                                                       0.0%    10   12.7  12.5   8.4  19.3   3.5
     2. AS6830   109.255.255.254                                                   0.0%    10   11.5  15.2   9.5  32.9   6.9
     3. AS6830   ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42)                   0.0%    10   13.5  14.6  10.6  20.4   3.1
     4. AS6830   ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110)                  0.0%    10   13.6  13.8   8.8  18.2   3.1
     5. AS1299   dln-b2-link.ip.twelve99.net (62.115.172.136)                      0.0%    10   11.3  13.7  10.3  17.7   2.6
     6. AS1299   ldn-bb1-link.ip.twelve99.net (62.115.120.100)                     0.0%    10   21.3  24.5  19.2  31.8   4.0
     7. AS1299   nyk-bb2-link.ip.twelve99.net (62.115.113.20)                      0.0%    10   99.5  96.3  92.8 101.2   2.8
     8. AS1299   ash-bb2-link.ip.twelve99.net (62.115.136.201)                     0.0%    10  100.0 100.6  97.6 106.0   2.7
     9. AS1299   ash-b1-link.ip.twelve99.net (62.115.143.121)                      0.0%    10  100.8 104.7 100.8 119.1   5.5
    10. AS1299   wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226)   0.0%    10   99.8 102.0  96.5 109.1   3.9
    11. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
    12. AS???    ???                                                              100.0    10    0.0   0.0   0.0   0.0   0.0
    13. AS14907  208.80.154.228                                                    0.0%    10  104.1  98.3  94.5 104.1   3.1


IPv6 route ok also:

   an-worker1147-test# show bgp ipv6 neighbors 2620:0:861:10d::253 advertised-routes 
   BGP table version is 3, local router ID is 208.80.154.228, vrf id 0
   Default local pref 100, local AS 64601
   Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                  i internal, r RIB-failure, S Stale, R Removed
   Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
   Origin codes:  i - IGP, e - EGP, ? - incomplete
   RPKI validation codes: V valid, I invalid, N Not found
   
      Network          Next Hop            Metric LocPrf Weight Path
   *> 2620:0:861:ed1a::4/128
                       ::                       0         32768 ?
   
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:10d::72 
   
   PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
     2620:0:861:ed1a::4/128
   *                         2620:0:861:10d::72   0                  64601 ?
   cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 2620:0:861:10d::1 rapid count 100 2620:0:861:10d::72 
   PING6(56=40+8+8 bytes) 2620:0:861:10d::1 --> 2620:0:861:10d::72
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   cathal@nbgw:~$ mtr -b -w -z -c 10 2620:0:861:ed1a::4
   Start: 2022-02-15T17:22:42+0000
   HOST: nbgw                                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS6939   tunnel650354.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:32c::1)   0.0%    10   32.2  31.5  22.6  61.3  11.3
     2. AS6939   e0-19.core2.lon2.he.net (2001:470:0:67::1)                           0.0%    10   28.8  29.7  23.7  45.3   6.7
     3. AS6939   100ge11-2.core1.lon2.he.net (2001:470:0:541::1)                      0.0%    10   25.8  39.4  19.9  93.0  25.3
     4. AS6939   100ge4-1.core1.nyc4.he.net (2001:470:0:2cf::2)                      10.0%    10   86.2  94.6  86.2 140.8  17.5
     5. AS6939   100ge11-1.core1.nyc5.he.net (2001:470:0:20a::2)                      0.0%    10   92.0  97.1  86.9 127.6  15.7
     6. AS???    ???                                                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     7. AS6939   100ge1-2.core1.ash1.he.net (2001:470:0:277::1)                      10.0%    10   95.5  99.1  90.0 116.7   9.4
     8. AS6939   xe-5-3-3-500.cr1-eqiad.wikimedia.org (2001:470:0:1c0::2)             0.0%    10   94.9  99.0  89.4 135.3  13.5
     9. AS14907  et-0-0-48-100.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:fe07::2)         0.0%    10   93.3 104.5  92.2 145.2  19.3
    10. AS14907  2620:0:861:ed1a::4                                                   0.0%    10   98.0  98.6  89.9 126.4  12.3

BGP Re-establishment if device peered to Anycast GW moved to new switch (i.e. VM live motion)

This will need to be re-visited when we have a Ganeti cluster in the new cage. Without that it could not be tested.

The idea of this test is to confirm that a VM, with a BGP peering to its default gateway, will restore the BGP session in reasonable time if it's migrated to a new hypervisor host. As the new hypervisor will be connected to a different switch, the VM will get TCP RST back from the new top-of-rack when it sends TCP packets relating to the its BGP connection to the old switch. At this point it should re-initiate the session with the new top-of-rack and traffic should begin to flow again.

This was tested with vQFX lab devices and interruptions were less than 10 seconds, so we can be confident there won't be any issue.

eBGP Peering in VRF to external device

This test simulates doing an eBGP peering from the switch fabric overlay over a normal routed link. In production an example would be the connections from Spine switch to core routers. Assumption is it works similar to peering to a device on a connected Vlan from an IRB, but need to validate.

Relevant Config

Config in the test was to our CR routers in Eqiad. Separate peerings were established over IPv4 and IPv6 to exchange routes of each type:

   cmooney@lsw1-e1-eqiad> show configuration interfaces et-0/0/48 
   description "Core: cr1-eqiad:et-1/0/2";
   vlan-tagging;
   mtu 9192;
   encapsulation flexible-ethernet-services;
   unit 100 {
       description "cr1-eqiad et-1/0/2.100 - Production VRF";
       vlan-id 100;
       family inet {
           address 10.66.0.9/31;
       }
       family inet6 {
           address 2620:0:861:fe07::2/64;
       }
   }
   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group EXTERNAL4  
   type external;
   hold-time 30;
   import DEFAULT4;
   family inet {
       unicast {
           prefix-limit {
               maximum 1000;
               teardown;
           }
       }
   }
   export EXT_OUT4;
   peer-as 14907;
   neighbor 10.66.0.8 {
       description cr1-eqiad;
   }
   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group EXTERNAL6    
   type external;
   hold-time 30;
   import DEFAULT6;
   family inet6 {
       unicast {
           prefix-limit {
               maximum 1000;
               teardown;
           }
       }
   }
   export EXT_OUT6;
   peer-as 14907;
   neighbor 2620:0:861:fe07::1 {
       description cr1-eqiad;
   }

IPv4

IPv4 peering works and routes exchanged:

   cmooney@lsw1-e1-eqiad> show bgp summary | match 14907 
   10.66.0.8             14907     106565     112060       0       6 1w4d 18:48:37 Establ
   2620:0:861:fe07::1       14907     106564     112039       0       4 1w4d 18:48:26 Establ
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.66.0.8 
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 0.0.0.0/0               10.66.0.8                               14907 I
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 advertising-protocol bgp 10.66.0.8 
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 10.64.130.0/24          Self                                    I
   * 10.64.131.0/24          Self                                    I
   * 10.64.132.0/24          Self                                    I
   * 10.64.134.0/24          Self                                    I
   * 10.64.138.0/24          Self                                    I
   * 10.64.142.0/24          Self                                    I
   * 208.80.154.226/32       Self                                    64600 ?
   * 208.80.154.228/32       Self                                    64601 ?
   

IPv6

   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:fe07::1    
   
   PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * ::/0                    2620:0:861:fe07::1                      14907 I
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 advertising-protocol bgp 2620:0:861:fe07::1 
   
   PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 2620:0:861:100::/64     Self                                    I
   * 2620:0:861:109::/64     Self                                    I
   * 2620:0:861:10a::/64     Self                                    I
   * 2620:0:861:10b::/64     Self                                    I
   * 2620:0:861:10d::/64     Self                                    I
   * 2620:0:861:114::/64     Self                                    I
     2620:0:861:ed1a::3/128
   *                         Self                                    64600 ?
     2620:0:861:ed1a::4/128
   *                         Self                                    64601 ?
     2620:0:861:fe07::/64
   *                         Self                                    I
     2620:0:861:fe08::/64
   *                         Self                                    I

BGP Route propagaton from unicast peer into EVPN and into remote VRF table

This test validates routes learnt from unicast BGP peers in the overlay are properly propagated as an EVPN type 5 routes between devices in the switch fabric, and pushed to the correct routing table/VRF on remote devices based on the parameters in the EVPN route.

Routes learnt on Spine1:

   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.66.0.8 
   
   PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 0.0.0.0/0               10.66.0.8                               14907 I
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:fe07::1    
   
   PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * ::/0                    2620:0:861:fe07::1                      14907 I

Routes learnt on Spine2:

   cmooney@lsw1-f1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.66.0.10     
   
   PRODUCTION.inet.0: 32 destinations, 35 routes (32 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 0.0.0.0/0               10.66.0.10                              14907 I
   cmooney@lsw1-f1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:fe08::1 
   
   PRODUCTION.inet6.0: 38 destinations, 42 routes (38 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * ::/0                    2620:0:861:fe08::1                      14907 I


These are exported as EVPN type 5 routes, announced and received by downstream LEAF switch lsw1-e3-eqiad:

   cmooney@lsw1-e3-eqiad> show route table bgp.evpn.0 aspath-regex ".* 14907$"    
   
   bgp.evpn.0: 79 destinations, 147 routes (79 active, 0 holddown, 0 hidden)
   + = Active Route, - = Last Active, * = Both
   
   5:10.64.128.3:5000::0::0.0.0.0::0/248               
                      *[BGP/170] 1w4d 19:04:01, localpref 100, from 10.64.128.3
                         AS path: 14907 I, validation-state: unverified
                       >  to 10.64.129.2 via et-0/0/54.0
                       [BGP/170] 1w4d 19:04:00, localpref 100, from 10.64.128.7
                         AS path: 14907 I, validation-state: unverified
                       >  to 10.64.129.2 via et-0/0/54.0
   5:10.64.128.7:5000::0::0.0.0.0::0/248               
                      *[BGP/170] 1w4d 23:37:43, localpref 100, from 10.64.128.7
                         AS path: 14907 I, validation-state: unverified
                       >  to 10.64.129.16 via et-0/0/55.0
                       [BGP/170] 1w4d 23:37:43, localpref 100, from 10.64.128.3
                         AS path: 14907 I, validation-state: unverified
                       >  to 10.64.129.16 via et-0/0/55.0
   5:10.64.128.3:5000::0::::::0/248               
                      *[BGP/170] 1w4d 19:03:49, localpref 100, from 10.64.128.3
                         AS path: 14907 I, validation-state: unverified
                       >  to 10.64.129.2 via et-0/0/54.0
                       [BGP/170] 1w4d 19:03:49, localpref 100, from 10.64.128.7
                         AS path: 14907 I, validation-state: unverified
                       >  to 10.64.129.2 via et-0/0/54.0
   5:10.64.128.7:5000::0::::::0/248               
                      *[BGP/170] 1w4d 23:37:39, localpref 100, from 10.64.128.7
                         AS path: 14907 I, validation-state: unverified
                       >  to 10.64.129.16 via et-0/0/55.0
                       [BGP/170] 1w4d 23:37:39, localpref 100, from 10.64.128.3
                         AS path: 14907 I, validation-state: unverified
                       >  to 10.64.129.16 via et-0/0/55.0


These are imported into the local EVPN table as shown here:

   cmooney@lsw1-e3-eqiad> show evpn ip-prefix-database prefix 0.0.0.0/0         
   L3 context: PRODUCTION
   
   EVPN->IPv4 Imported Prefixes
   Prefix                                       Etag
   0.0.0.0/0                                    0       
     Route distinguisher    VNI/Label  Router MAC         Nexthop/Overlay GW/ESI
     10.64.128.3:5000       3005000    a4:e1:1a:81:3a:80  10.64.128.3
     10.64.128.7:5000       3005000    a4:e1:1a:81:9e:80  10.64.128.7
   cmooney@lsw1-e3-eqiad> show evpn ip-prefix-database prefix ::/0              
   L3 context: PRODUCTION
   
   EVPN->IPv6 Imported Prefixes
   Prefix                                       Etag
   ::/0                                         0       
     Route distinguisher    VNI/Label  Router MAC         Nexthop/Overlay GW/ESI
     10.64.128.3:5000       3005000    a4:e1:1a:81:3a:80  10.64.128.3
     10.64.128.7:5000       3005000    a4:e1:1a:81:9e:80  10.64.128.7


The IPv4 route is imported into the routing table for the 'PRODUCTION' routing-instance, which is what we expect based on the configuration:

   cmooney@lsw1-e3-eqiad> show route table PRODUCTION.inet.0 0.0.0.0/0 exact                
   
   PRODUCTION.inet.0: 30 destinations, 36 routes (30 active, 0 holddown, 0 hidden)
   @ = Routing Use Only, # = Forwarding Use Only
   + = Active Route, - = Last Active, * = Both
   
   0.0.0.0/0          @[EVPN/170] 1w4d 22:56:36
                       >  to 10.64.129.2 via et-0/0/54.0
                       [EVPN/170] 1w5d 03:30:18
                       >  to 10.64.129.16 via et-0/0/55.0
                      #[Multipath/255] 1w4d 22:56:36, metric2 8
                       >  to 10.64.129.2 via et-0/0/54.0
                          to 10.64.129.16 via et-0/0/55.0


This is also true for IPv6:

   cmooney@lsw1-f2-eqiad> show route table PRODUCTION.inet6.0 ::/0 exact 
   
   PRODUCTION.inet6.0: 37 destinations, 44 routes (37 active, 0 holddown, 0 hidden)
   @ = Routing Use Only, # = Forwarding Use Only
   + = Active Route, - = Last Active, * = Both
   
   ::/0               @[EVPN/170] 1w5d 22:23:30
                       >  to 10.64.129.8 via et-0/0/54.0
                       [EVPN/170] 1w5d 19:39:18
                       >  to 10.64.129.22 via et-0/0/55.0
                      #[Multipath/255] 1w5d 19:39:18, metric2 8
                       >  to 10.64.129.8 via et-0/0/54.0
                          to 10.64.129.22 via et-0/0/55.0

BGP Route propagation to external hosts from VRF [IPv4]

The purpose of this test is to show that routes originating in the EVPN overlay / VRF are properly advertised to external peers.

The check is mainly that those routes are announced from the Spine switches to our CRs correctly.

IPv4

Relevant Config

   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group EXTERNAL4  
   type external;
   hold-time 30;
   import DEFAULT4;
   family inet {
       unicast {
           prefix-limit {
               maximum 1000;
               teardown;
           }
       }
   }
   export EXT_OUT4;
   peer-as 14907;
   neighbor 10.66.0.8 {
       description cr1-eqiad;
   }
   cmooney@lsw1-e1-eqiad> show configuration policy-options policy-statement EXT_OUT4        
   /* Controls export of routes from VRF to external peers */
   term LVS {
       from {
           protocol [ bgp evpn ];
           as-path LOCAL_LVS;
       }
       then accept;
   }
   term K8_ANYCAST {
       from {
           protocol [ bgp evpn ];
           as-path LOCAL_K8_ANYCAST;
       }
       then accept;
   }
   term NETWORKS {
       from {
           protocol [ direct evpn ];
           route-filter 0.0.0.0/0 prefix-length-range /0-/29;
       }
       then accept;
   }
   then reject;


Results:

Top-of-rack switch subnets are propagated as expected (and, somewhat importantly, /32 host routes from EVPN type 2 sources / ARP snooping are not announced):

   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 advertising-protocol bgp 10.66.0.8    
   
   PRODUCTION.inet.0: 30 destinations, 31 routes (30 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 10.64.130.0/24          Self                                    I
   * 10.64.131.0/24          Self                                    I
   * 10.64.132.0/24          Self                                    I
   * 10.64.134.0/24          Self                                    I
   * 10.64.138.0/24          Self                                    I
   * 10.64.142.0/24          Self                                    I
   * 208.80.154.226/32       Self                                    64600 ?
   * 208.80.154.229/32       Self                                    64600 ?

IPv6

   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group EXTERNAL6    
   type external;
   hold-time 30;
   import DEFAULT6;
   family inet6 {
       unicast {
           prefix-limit {
               maximum 1000;
               teardown;
           }
       }
   }
   export EXT_OUT6;
   peer-as 14907;
   neighbor 2620:0:861:fe07::1 {
       description cr1-eqiad;
   }
   cmooney@lsw1-e1-eqiad> show configuration policy-options policy-statement EXT_OUT6                      
   /* Controls export of routes from VRF to external peers. */
   term LVS {
       from {
           protocol [ bgp evpn ];
           as-path LOCAL_LVS;
       }
       then accept;
   }
   term EVPN_K8_ANYCAST {
       from {
           protocol [ bgp evpn ];
           as-path LOCAL_K8_ANYCAST;
       }
       then accept;
   }
   term NETWORKS {
       from {
           protocol [ direct evpn ];
           route-filter ::/0 prefix-length-range /0-/125;
       }
       then accept;
   }
   then reject;
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 advertising-protocol bgp 2620:0:861:fe07::1  
   
   PRODUCTION.inet6.0: 36 destinations, 37 routes (36 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 2620:0:861:100::/64     Self                                    I
   * 2620:0:861:109::/64     Self                                    I
   * 2620:0:861:10a::/64     Self                                    I
   * 2620:0:861:10b::/64     Self                                    I
   * 2620:0:861:114::/64     Self                                    I
     2620:0:861:ed1a::3/128
   *                         Self                                    64600 ?
     2620:0:861:ed1a::6/128
   *                         Self                                    64600 ?
     2620:0:861:fe07::/64
   *                         Self                                    I
     2620:0:861:fe08::/64
   *                         Self                                    I

BGP Peering to external host from VRF using local-as 14907

If possible we want to avoid having to configure end-hosts with the ASN of the top-of-rack switch they are connected to. Instead our preference is to "fake" the AS used for peering from the switch side, always using the WMF public AS 14907, so that this piece of the configuration can be kept constant on the server side.

In order to do this the switch needs to support some form of "local-as" override. We need to be careful that:

  • BGP session establishes.
  • AS14907 is not placed in the AS-path when prefixes are propagated onwards (i.e. sent to CRs).
  • The device using fake 'local-as' 14907 can still establish and learn routes from an eBGP peer using that ASN (i.e. learnt from CRs).
  • If announcing routes over the BGP session only the the configured 'local-as' is included in the AS-path, the switch's default ASN is not present.

Relevant Config

The previous configuration for the PyBal group was modified with the 'local-as' configuration as shown:

   cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group PyBal             
   type external;
   hold-time 30;
   import LVS_IMPORT;
   family inet {
       unicast {
           prefix-limit {
               maximum 1000;
               teardown 20;
           }
       }
   }
   family inet6 {
       unicast {
           prefix-limit {
               maximum 1000;
               teardown 20;
           }
       }
   }
   export NONE;
   peer-as 64600;
   local-as 14907 loops 2 private no-prepend-global-as;
   neighbor 10.64.130.10 {
       description ms-fe1012-test;
   }


To break down the meaning of these commands:

   local-as:  Determines what AS to use for session
   loops 2:   With 'local-as' set to 14907 the switch will not accept that AS in a received path, "loops 2" allows routes with 1 instance of it in.
   private:   This tells JunOS the 'local-as' is only relevant to the peering, it should use its normal ASN when propagating routes learnt on it.
   no-prepend-global-as:   This tells JunOS to not put its own, globally-configured ASN, into the as-path if announcing routes to this peer.

Results

Adjacency forms from test-host with peer-as set to 14907:

   ms-fe1012-test# show bgp ipv4 unicast summary 
   BGP router identifier 208.80.154.226, local AS number 64600 vrf-id 0
   BGP table version 19
   RIB entries 3, using 552 bytes of memory
   Peers 1, using 723 KiB of memory
   
   Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
   10.64.130.1     4      14907      3256      2968        0    0    0 06:46:07            0        1 N/A

The route being announced is accepted and learnt with as-path of the "PyBal" peer (64600):

   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.64.130.10 
   
   PRODUCTION.inet.0: 30 destinations, 31 routes (30 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 208.80.154.226/32       10.64.130.10         0                  64600 ?
  

This is correctly being advertised externally to the CR, which we are peering to on its AS 14907:

   cmooney@lsw1-e1-eqiad> show bgp summary | match 14907 
   10.66.0.8             14907       2147       2266       0       0     5:41:29 Establ
   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 advertising-protocol bgp 10.66.0.8 | match 208.80.154.226     
   * 208.80.154.226/32       Self                                    64600 ?

On the CR the route is learnt with the switch's ASN followed by the PyBal one, as desired:

   cmooney@re0.cr1-eqiad> show route receive-protocol bgp 10.66.0.9 | match 208.80.154.226 
   * 208.80.154.226/32       10.66.0.9                               64810 64600 ?

On the switch we still learn and accept the route sent by the CR (with AS14907 in the path):

   cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.66.0.8                               
   
   PRODUCTION.inet.0: 30 destinations, 31 routes (30 active, 0 holddown, 0 hidden)
     Prefix		  Nexthop	       MED     Lclpref    AS path
   * 0.0.0.0/0               10.66.0.8                               14907 I

Inernal routing from device connected to access vlan

This tests validates connectivity between hosts in the new racks and devices connected to VC-based switches in other rows.

IPv4

   root@elastic1093-test:~# mtr --address 10.64.132.2 -b -w -c 10 10.64.16.8
   Start: 2022-02-17T21:13:05+0000
   HOST: elastic1093-test                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
     1.|-- irb-1033.lsw1-e3-eqiad.eqiad.wmnet (10.64.132.1)  0.0%    10    4.4   4.7   2.0   8.4   2.4
     2.|-- irb-1035.lsw1-f1-eqiad.eqiad.wmnet (10.64.134.1)  0.0%    10    1.3   3.4   0.9   6.2   1.9
     3.|-- et-1-0-2-100.cr2-eqiad.eqiad.wmnet (10.66.0.10)   0.0%    10    0.3   1.0   0.3   6.1   1.8
     4.|-- phab1001.eqiad.wmnet (10.64.16.8)                 0.0%    10    0.2   0.3   0.2   0.3   0.0

IPv6

   root@elastic1093-test:~# mtr --address 2620:0:861:10b:10:64:132:2 -b -w -c 10  2620:0:861:102:10:64:16:8
   Start: 2022-02-17T21:12:33+0000
   HOST: elastic1093-test                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
     1.|-- irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1)   0.0%    10    0.6   0.5   0.5   0.6   0.0
     2.|-- irb-1039.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:100::1)   0.0%    10    0.6   0.6   0.5   0.7   0.1
     3.|-- et-1-0-2-100.cr1-eqiad.eqiad.wmnet (2620:0:861:fe07::1)  0.0%    10    0.4   0.6   0.4   1.8   0.4
     4.|-- phab1001.eqiad.wmnet (2620:0:861:102:10:64:16:8)         0.0%    10    0.3   0.3   0.3   0.3   0.0

External routing from device connected to access vlan

This tests validates connectivity between hosts in the new racks and devices connected to VC-based switches in other rows.

IPv4

   root@elastic1093-test:~# mtr -4 --address 208.80.154.229 -z -b -w -c 10 www.ietf.org
   Start: 2022-02-17T21:16:38+0000
   HOST: elastic1093-test                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS???    irb-1033.lsw1-e3-eqiad.eqiad.wmnet (10.64.132.1)   0.0%    10    7.1   5.2   0.7  13.3   3.9
     2. AS???    irb-1035.lsw1-f1-eqiad.eqiad.wmnet (10.64.134.1)   0.0%    10    1.8   4.5   1.4   7.8   2.3
     3. AS???    et-1-0-2-100.cr2-eqiad.eqiad.wmnet (10.66.0.10)    0.0%    10    0.4   0.5   0.4   0.6   0.1
     4. AS???    13335.ash.equinix.com (206.126.237.30)            60.0%    10    1.8   7.7   1.8  13.3   6.4
     5. AS13335  172.70.172.2                                       0.0%    10    1.7   3.0   1.2  11.5   3.2
     6. AS13335  104.16.45.99                                       0.0%    10    1.2   1.2   1.1   1.3   0.0

IPv6

   root@elastic1093-test:~# mtr -6 --address 2620:0:861:ed1a::6 -z -b -w -c 10 www.ietf.org
   Start: 2022-02-17T21:17:12+0000
   HOST: elastic1093-test                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS14907  irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1)    0.0%    10    0.6   0.5   0.4   0.6   0.1
     2. AS14907  irb-1039.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:100::1)    0.0%    10    7.9   5.2   0.5  40.2  12.5
     3. AS14907  et-1-0-2-100.cr1-eqiad.eqiad.wmnet (2620:0:861:fe07::1)   0.0%    10    0.6   2.4   0.4  18.9   5.8
     4. AS14907  ae0.cr2-eqiad.wikimedia.org (2620:0:861:fe00::2)          0.0%    10    0.3   0.4   0.3   0.5   0.0
     5. AS???    13335.ash.equinix.com (2001:504:0:2:0:1:3335:1)          60.0%    10    1.3   5.4   1.3  17.0   7.8
     6. AS13335  2400:cb00:350:3::                                         0.0%    10    1.3   4.2   1.1  15.2   5.3
     7. AS13335  2606:4700::6810:2c63                                      0.0%    10    0.4   0.5   0.4   0.8   0.1


Worth observing that hop 2 is different in each case, as lsw1-e3 has two upstream "spines" (lsw1-e1 and lsw1-f1), and load-balances traffic across them.

ARP Supression

ARP suppression is a feature in EVPN layer-2 networks, which aims to minimise the amount of layer-2 broadcast traffic being sent, and thus support larger layer-2 segments. It is made possible through the inclusion of IP as well as MAC information in EVPN type 2 routes. The IP data is inserted by switches based on either on local ARP tables or snooping on traffic. As this is distributed in BGP/EVPN, devices become aware of remote MAC/IP bindings, and can answer ARP requests from their own hosts if they have the answer.

Checks

Ping Anycast GW 10.64.134.1 of lsw1-e1 from directly connected 10.64.131.72.

   root@an-worker1147-test:~# ping -I 10.64.134.72 -c 2 10.64.134.1
   PING 10.64.134.1 (10.64.134.1) from 10.64.134.72 : 56(84) bytes of data.
   64 bytes from 10.64.134.1: icmp_seq=1 ttl=64 time=5.28 ms
   64 bytes from 10.64.134.1: icmp_seq=2 ttl=64 time=2.71 ms

ARP entry is populated on lsw1-e1 as desired:

   cmooney@lsw1-e1-eqiad> show arp no-resolve expiration-time | match 10.64.134.72 
   e4:3d:1a:54:14:45 10.64.134.72    irb.1035 [xe-0/0/6.0]           permanent remote

In turn an EVPN route is learnt on lsw1-f1 containing this information:

   cmooney@lsw1-f1-eqiad> show route protocol bgp table bgp.evpn.0 terse | match 10.64.134.72         
       2:10.64.128.3:64810::2001035::e4:3d:1a:54:14:45::10.64.134.72/304 MAC/IP 

EVPN database shows it in more detail:

   cmooney@lsw1-e1-eqiad> show evpn database extensive mac-address e4:3d:1a:54:14:45  
   Instance: default-switch
   
   VN Identifier: 2001035, MAC address: e4:3d:1a:54:14:45
     State: 0x0
     Source: xe-0/0/6.0, Rank: 1, Status: Active
       Mobility sequence number: 0 (minimum origin address 10.64.128.3)
       Timestamp: Feb 17 21:38:12.700143 (0x620ec044)
       State: <Local-MAC-Only Local-To-Remote-Adv-Allowed>
       MAC advertisement route status: Created
       IP address: 10.64.134.72
       Flags: <Local-Adv>
         L3 route: 10.64.134.72/32, L3 context: PRODUCTION (irb.1035)
       IP address: 2620:0:861:10d::72
       Flags: <Local-Adv>
         L3 route: 2620:0:861:10d::72/128, L3 context: PRODUCTION (irb.1035)
       IP address: fe80::e63d:1aff:fe54:1445
       Flags: <Local-Adv>
         L3 route: fe80::e63d:1aff:fe54:1445/128, L3 context: PRODUCTION (irb.1035)
       History db: 
         Time                       Event
         Feb 17 12:36:30.139 2022   xe-0/0/6.0 : Created
         Feb 17 12:36:30.139 2022   Updating output state (change flags 0x1 <ESI-Added>)
         Feb 17 12:36:30.139 2022   Active ESI changing (not assigned -> xe-0/0/6.0)
         Feb 17 12:36:30.139 2022   xe-0/0/6.0 : Updating output state (change flags 0x200 <IP-Added>)
         Feb 17 12:37:40.161 2022   xe-0/0/6.0 : Updating output state (change flags 0x200 <IP-Added>)
         Feb 17 21:35:36.597 2022   xe-0/0/6.0 : 10.64.134.72 Selected IRB interface nexthop
         Feb 17 21:35:36.597 2022   xe-0/0/6.0 : 2620:0:861:10d::72 Selected IRB interface nexthop
         Feb 17 21:38:12.700 2022   xe-0/0/6.0 : Updating output state (change flags 0x200 <IP-Added>)
         Feb 17 21:38:12.700 2022   xe-0/0/6.0 : fe80::e63d:1aff:fe54:1445 Selected IRB interface nexthop

Ultimately this causes an ARP entry to be added on the lsw1-f1 binding the same MAC/IP:

   cmooney@lsw1-f1-eqiad> show arp no-resolve expiration-time | match 10.64.134.72   
   e4:3d:1a:54:14:45 10.64.134.72    irb.1035 [vtep.32769]           permanent remote


Test

Now that the IP/MAC info for 10.64.134.72 is populated across the network we need to check what happens when we send an ARP request for this IP from a host on the same Vlan, but connected to a different switch.

To begin I connected a host port with no IP address configured to a port on lsw1-f1, port was configured as an access port in Vlan 1035 / private1-f1-eqiad. Running 'date' a lot so the timing is clearer, I first added an IP on the hosts interface, making this the first moment it could participate in ARP:

   root@an-worker1148-test:/etc/systemd# date && ip addr add 10.64.134.11/24 dev eno2np1
   Thu 17 Feb 22:54:59 GMT 2022

I then pinged the Anycast GW IP (which will be lsw1-f1 as that's what we're connected to):

   root@an-worker1148-test:/etc/systemd# date && ping 10.64.134.1
   Thu 17 Feb 22:55:29 GMT 2022
   PING 10.64.134.1 (10.64.134.1) 56(84) bytes of data.
   64 bytes from 10.64.134.1: icmp_seq=1 ttl=64 time=98.0 ms
   64 bytes from 10.64.134.1: icmp_seq=2 ttl=64 time=1.14 ms

And now ping for the other host, connected to remote switch lsw1-e1:

   root@an-worker1148-test:/etc/systemd# date && ping 10.64.134.72
   Thu 17 Feb 22:55:33 GMT 2022
   PING 10.64.134.72 (10.64.134.72) 56(84) bytes of data.
   64 bytes from 10.64.134.72: icmp_seq=1 ttl=64 time=14.5 ms
   64 bytes from 10.64.134.72: icmp_seq=2 ttl=64 time=0.230 ms

Looking over packet captures running on this device, it sent ARPs as expected for both IPs pinged, and got answers for both:

   root@an-worker1148-test:~# date && tcpdump -i eno2np1 -l -nn arp  && date
   Thu 17 Feb 22:54:58 GMT 2022
   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   listening on eno2np1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   22:55:29.065596 ARP, Request who-has 10.64.134.1 tell 10.64.134.11, length 28
   22:55:29.075964 ARP, Reply 10.64.134.1 is-at 00:00:5e:11:fa:ce, length 46
   22:55:33.324530 ARP, Request who-has 10.64.134.72 tell 10.64.134.11, length 28
   22:55:33.338783 ARP, Reply 10.64.134.72 is-at e4:3d:1a:54:14:45, length 46

A tcpdump on 10.64.134.72 reveals, however, that no ARP request hit the server from 10.64.134.11. The first packets we see from the other IP are the ICMP packets:

   root@an-worker1147-test:/etc/systemd# date && tcpdump -i eno2np1.1035 -l -nn host 10.64.134.11 or arp && date 
   Thu 17 Feb 22:54:54 GMT 2022
   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   listening on eno2np1.1035, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   22:55:33.338005 IP 10.64.134.11 > 10.64.134.72: ICMP echo request, id 16418, seq 1, length 64
   22:55:33.338049 IP 10.64.134.72 > 10.64.134.11: ICMP echo reply, id 16418, seq 1, length 64
   22:55:34.325397 IP 10.64.134.11 > 10.64.134.72: ICMP echo request, id 16418, seq 2, length 64
   22:55:34.325424 IP 10.64.134.72 > 10.64.134.11: ICMP echo reply, id 16418, seq 2, length 64

We can thus conclude the ARP request an-worker1147 sent from 10.64.134.71 was not broadcast to all devices across the network, and instead lsw1-f1 generated the appropriate response when it received it, using the information from EVPN. This is an adequate, if not a very sophisticated, way to verify this. Unfortunately the QFX doesn't give any stats or counters on it, and Juniper's own test instructions are similarly based on doing packet captures.

ND Suppression

ND suppression in IPv6 is very similar to the concept of ARP suppression in IPv4. Once one switch has the IP/MAC binding it's included in EVPN type 2 routes, which mean the whole network should have this info. If other hosts perform neighbor discovery for a given IP, the packets are not multicast in the Vlan but the switches should generate a fake response, with the correct info learnt from EVPN, reducing the level of BUM traffic.

Configuration / vlan is the same as before.

Results

Again 'date' used to make the sequence clearer. First the IP address is added to the interface:

   root@an-worker1148-test:~# date && ip addr add 2620:0:861:10d::11/64 dev eno2np1
   Thu 17 Feb 23:34:12 GMT 2022

Then we ping the Anycast GW (will hit lsw1-f1 where we're connected):

   root@an-worker1148-test:~# date && ping 2620:0:861:10d::1
   Thu 17 Feb 23:34:22 GMT 2022
   PING 2620:0:861:10d::1(2620:0:861:10d::1) 56 data bytes
   64 bytes from 2620:0:861:10d::1: icmp_seq=1 ttl=64 time=12.2 ms
   64 bytes from 2620:0:861:10d::1: icmp_seq=2 ttl=64 time=0.491 ms

Lastly we ping an-worker1147, connected in the same Vlan on a remote switch:

   root@an-worker1148-test:~# date && ping 2620:0:861:10d::72
   Thu 17 Feb 23:34:29 GMT 2022
   PING 2620:0:861:10d::72(2620:0:861:10d::72) 56 data bytes
   64 bytes from 2620:0:861:10d::72: icmp_seq=1 ttl=64 time=21.2 ms
   64 bytes from 2620:0:861:10d::72: icmp_seq=2 ttl=64 time=0.232 ms

A packet capture on the device doing the pings shows several ND requests, including for an-worker1148:

   root@an-worker1148-test:~# date && tcpdump -i eno2np1 -l -nn icmp6
   Thu 17 Feb 23:33:40 GMT 2022
   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   listening on eno2np1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   23:34:12.918598 IP6 :: > ff02::1:ff00:11: ICMP6, neighbor solicitation, who has 2620:0:861:10d::11, length 32
   23:34:22.706295 IP6 2620:0:861:10d::11 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2620:0:861:10d::1, length 32
   23:34:22.710987 IP6 2620:0:861:10d::1 > 2620:0:861:10d::11: ICMP6, neighbor advertisement, tgt is 2620:0:861:10d::1, length 32
   23:34:22.711020 IP6 2620:0:861:10d::11 > 2620:0:861:10d::1: ICMP6, echo request, id 10401, seq 1, length 64
   23:34:22.718416 IP6 2620:0:861:10d::1 > 2620:0:861:10d::11: ICMP6, echo reply, id 10401, seq 1, length 64
   23:34:23.707671 IP6 2620:0:861:10d::11 > 2620:0:861:10d::1: ICMP6, echo request, id 10401, seq 2, length 64
   23:34:23.708132 IP6 2620:0:861:10d::1 > 2620:0:861:10d::11: ICMP6, echo reply, id 10401, seq 2, length 64
   23:34:29.649454 IP6 2620:0:861:10d::11 > ff02::1:ff00:72: ICMP6, neighbor solicitation, who has 2620:0:861:10d::72, length 32
   23:34:29.659961 IP6 2620:0:861:10d::72 > 2620:0:861:10d::11: ICMP6, neighbor advertisement, tgt is 2620:0:861:10d::72, length 32
   23:34:29.659993 IP6 2620:0:861:10d::11 > 2620:0:861:10d::72: ICMP6, echo request, id 38872, seq 1, length 64
   23:34:29.670614 IP6 2620:0:861:10d::72 > 2620:0:861:10d::11: ICMP6, echo reply, id 38872, seq 1, length 64
   23:34:30.650792 IP6 2620:0:861:10d::11 > 2620:0:861:10d::72: ICMP6, echo request, id 38872, seq 2, length 64
   23:34:30.650994 IP6 2620:0:861:10d::72 > 2620:0:861:10d::11: ICMP6, echo reply, id 38872, seq 2, length 64

Capture on an-worker1148 shows, however, that the ND message never hit it:

   root@an-worker1147-test:~# tcpdump -i eno2np1.1035 -l -nn icmp6
   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   listening on eno2np1.1035, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   23:34:29.656794 IP6 2620:0:861:10d::11 > 2620:0:861:10d::72: ICMP6, echo request, id 38872, seq 1, length 64
   23:34:29.667235 IP6 2620:0:861:10d::72 > 2620:0:861:10d::11: ICMP6, echo reply, id 38872, seq 1, length 64
   23:34:30.647585 IP6 2620:0:861:10d::11 > 2620:0:861:10d::72: ICMP6, echo request, id 38872, seq 2, length 64
   23:34:30.647613 IP6 2620:0:861:10d::72 > 2620:0:861:10d::11: ICMP6, echo reply, id 38872, seq 2, length 64

Similar to the ARP test this shows that the switches are generating responses to ND requests locally, reducing BUM traffic in the Vlan.

DHCP Relay & Option 82 insertion

We format DHCP requests from end servers over unicast IPv4 to our 'install' servers in each datacenter. Those machines run ISC's dhcpd software and return IP addresses to end hosts if configured to do so for the specific end host. Determination of which host is requesting is based on the "option 82" information present in the DHCP DISCOVER messages from the host. This information is inserted by the top-of-rack switch when the packet goes through it, allowing this to take place.

Relevant Config

In our case the DHCP requests will arrive from IRB interface in an overlay VRF/routing-instance, and be relayed to the install server via the same instance. So the configuration is done in that part:

   cmooney@lsw1-e3-eqiad> show configuration routing-instances PRODUCTION forwarding-options               
   dhcp-relay {
       relay-option-82 {
           circuit-id {
               prefix {
                   host-name;
               }
           }
       }
       server-group {
           INSTALL-SERVER {
               208.80.154.32;
           }
       }
       group DHCP-RELAY {
           active-server-group INSTALL-SERVER;
           interface irb.1033;
           interface irb.1041;
       }
   }


Results

We can then send a DHCP request from a connected host, for instance elastic1093-test connected to lsw1-e3 xe-0/0/21.


DHCP DISCOVER is received on install1003 shortly after, with option 82 information populated as expected:

   cmooney@install1003:~$ sudo tcpdump -i ens5 -l -p -vvv -nn port 67 or port 68 and not src host 208.80.154.199
   tcpdump: listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes
   23:56:56.244662 IP (tos 0x0, ttl 62, id 6901, offset 0, flags [none], proto UDP (17), length 408)
       10.64.132.1.67 > 208.80.154.32.67: [udp sum ok] BOOTP/DHCP, Request from e4:3d:1a:cb:5c:21, length 380, hops 1, xid 0xc24c6629, Flags [none] (0x0000)
   	  Gateway-IP 10.64.132.1
   	  Client-Ethernet-Address e4:3d:1a:cb:5c:21
   	  Vendor-rfc1048 Extensions
   	    Magic Cookie 0x63825363
   	    DHCP-Message Option 53, length 1: Discover
   	    Hostname Option 12, length 16: "elastic1093-test"
   	    Parameter-Request Option 55, length 13: 
   	      Subnet-Mask, BR, Time-Zone, Default-Gateway
   	      Domain-Name, Domain-Name-Server, Option 119, Hostname
   	      Netbios-Name-Server, Netbios-Scope, MTU, Classless-Static-Route
   	      NTP
   	    Agent-Information Option 82, length 78: 
   	      Circuit-ID SubOption 1, length 43: lsw1-e3-eqiad:xe-0/0/21.0:private1-e3-eqiad
   	      Unknown SubOption 12, length 31: 
   		0x0000:  0002 0000 0000 0583 0100 0000 6134 3a65
   		0x0010:  313a 3161 3a38 313a 6435 3a38 3000 00
   	    END Option 255, length 0
   	    PAD Option 0, length 0, occurs 23

IPv6 Router Advertisement Generation

Although we statically configure end-host IPs in /etc/network/interfaces, the IPv6 address that is added there needs to be derived on a given host from a SLAAC-assigned address (giving us the subnet) and the IPv4 address (which we use as host portion of v6 address).

IPv6 router advertisements must be generated on all IRB/Vlan interfaces and sent to end hosts so they can configure an IPv6 global unicast address via SLAAC.

Relevant Config

   cmooney@lsw1-e3-eqiad> show configuration interfaces irb.1033 
   description private1-e3-eqiad;
   family inet {
       address 10.64.132.1/24;
   }
   family inet6 {
       address 2620:0:861:10b::1/64;
   }
   cmooney@lsw1-e3-eqiad> show configuration protocols router-advertisement     
   interface irb.1033 {
       max-advertisement-interval 30;
       default-lifetime 600;
       prefix 2620:0:861:10b::1/64;
   }

Results

RAs are received on end hosts connected to Vlan1033 as expected:

   cmooney@elastic1093:~$ sudo tcpdump -e -v -l -nn -i enp59s0f0np0 icmp6
   tcpdump: listening on enp59s0f0np0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   13:28:55.906194 a4:e1:1a:81:d5:80 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::a6e1:1a04:981:d580 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 56
   	hop limit 64, Flags [none], pref medium, router lifetime 600s, reachable time 0ms, retrans timer 0ms
   	  source link-address option (1), length 8 (1): a4:e1:1a:81:d5:80
   	  prefix info option (3), length 32 (4): 2620:0:861:10b::/64, Flags [onlink, auto], valid time 2592000s, pref. time 604800s

DHCP Relay On Stretched Vlan / IRB with Anycast GW

Similar to in the previous case, we send a DHCP request from an-worker1148, connected to lsw1-f1 on port xe-0/0/6.

IRB is configured with Anycast GW config like this:

   cmooney@lsw1-f1-eqiad> show configuration interfaces irb.1035   
   virtual-gateway-accept-data;
   description private1-f1-eqiad;
   family inet {
       address 10.64.134.254/24 {
           preferred;
           virtual-gateway-address 10.64.134.1;
       }
   }
   family inet6 {
       address 2620:0:861:10d::254/64 {
           preferred;
           virtual-gateway-address 2620:0:861:10d::1;
       }
   }
   virtual-gateway-v4-mac 00:00:5e:11:fa:ce;
   virtual-gateway-v6-mac 00:00:5e:11:fa:ce;


When we run the DHCP client on an-worker1148 we see this on the install server. Worth noting in the source IP is lsw1-e1-eqiad's unique IP on the subnet, rather than the VIP shared by all devices (this is important to ensure replies go back to correct device):

   cmooney@install1003:~$ sudo tcpdump -i ens5 -l -p -vvv -nn net 10.64.134.0/24
   tcpdump: listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes
   00:09:47.764115 IP (tos 0x0, ttl 63, id 23228, offset 0, flags [none], proto UDP (17), length 407)
       10.64.134.254.67 > 208.80.154.32.67: [udp sum ok] BOOTP/DHCP, Request from e4:3d:1a:54:ab:a7, length 379, hops 1, xid 0x8bee627f, Flags [none] (0x0000)
   	  Gateway-IP 10.64.134.254
   	  Client-Ethernet-Address e4:3d:1a:54:ab:a7
   	  Vendor-rfc1048 Extensions
   	    Magic Cookie 0x63825363
   	    DHCP-Message Option 53, length 1: Discover
   	    Hostname Option 12, length 18: "an-worker1148-test"
   	    Parameter-Request Option 55, length 13: 
   	      Subnet-Mask, BR, Time-Zone, Default-Gateway
   	      Domain-Name, Domain-Name-Server, Option 119, Hostname
   	      Netbios-Name-Server, Netbios-Scope, MTU, Classless-Static-Route
   	      NTP
   	    Agent-Information Option 82, length 77: 
   	      Circuit-ID SubOption 1, length 42: lsw1-f1-eqiad:xe-0/0/6.0:private1-f1-eqiad
   	      Unknown SubOption 12, length 31: 
   		0x0000:  0002 0000 0000 0583 0100 0000 6134 3a65
   		0x0010:  313a 3161 3a38 313a 3965 3a38 3000 00
   	    END Option 255, length 0
   	    PAD Option 0, length 0, occurs 21


NOTE: In addition to the above packet, the install server also received a relayed DHCP request from lsw1-e1-eqiad, the only other switch in the fabric with Vlan1035 (private1-f1-eqiad) configured, also with an Anycast GW.

What happens is the DHCP discover, as well as being relayed to the install server by the receiving switch, is also processed as a regular broadcast within the Vlan. When it is demuxed by other switches it ultimately hits their equivalent IRB interface, triggering the same DHCP relay config. So the remote switches also end up relaying the packet, causing multiple to hit the install server. They insert different Option 82 info, however, which won't ever match a config block on the install server:

   Circuit-ID SubOption 1, length 42: lsw1-e1-eqiad:'vtep.32770':private1-f1-eqiad

So despite this traffic being unwanted, it won't cause any problems. Having looked briefly it does not seem to be possible to prevent this, but we'll will have limited, if any, stretched Vlans with Anycast GWs, so it's not a big deal.

IP Filters on Routed interface

This test is to validate that IP packets are correctly filtered on direct routed links (i.e. non-IRB interfaces) in a routing-instance/VRF.

IPv4

Relevant Config

New firewall filter was added to both Spine layer devices as follows:

   cmooney@lsw1-e1-eqiad> show configuration firewall family inet filter BLOCK_MS_FE1012 
   term BLOCK {
       from {
           destination-address {
               208.80.154.226/32;
           }
       }
       then {
           log;
           reject;
       }
   }
   term ALLOW {
       then accept;
   }
   cmooney@lsw1-f1-eqiad> show configuration firewall family inet filter BLOCK_MS_FE1012 
   term BLOCK {
       from {
           destination-address {
               208.80.154.226/32;
           }
       }
       then {
           log;
           reject;
       }
   }
   term ALLOW {
       then accept;
   }


Test

The above filter was applied on each Spine as an input filter on the link from CR routers. The filter is applied both sides as traffic may arrive in via either CR router and thus we want to filter in both places to be sure to catch the packets.

   cmooney@lsw1-e1-eqiad# show | compare 
   [edit interfaces et-0/0/48 unit 100 family inet]
   +       filter {
   +           input BLOCK_MS_FE1012;
   +       }
   
   cmooney@lsw1-f1-eqiad# show | compare 
   [edit interfaces et-0/0/48 unit 100 family inet]
   +       filter {
   +           input BLOCK_MS_FE1012;
   +       }


We then kick off a ping from this IP, and once running we commit the above changes:

   root@ms-fe1012-test:~# ping -I 208.80.154.226 1.1.1.1
   PING 1.1.1.1 (1.1.1.1) from 208.80.154.226 : 56(84) bytes of data.
   64 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=0.504 ms
   64 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=0.562 ms
   64 bytes from 1.1.1.1: icmp_seq=3 ttl=59 time=0.564 ms
   64 bytes from 1.1.1.1: icmp_seq=4 ttl=59 time=0.530 ms
   64 bytes from 1.1.1.1: icmp_seq=5 ttl=59 time=0.602 ms
   ^C
   --- 1.1.1.1 ping statistics ---
   14 packets transmitted, 5 received, 64.2857% packet loss, time 13289ms
   rtt min/avg/max/mdev = 0.504/0.552/0.602/0.033 ms

Looking at device F1 we can see the packets were dropped as expected:

   cmooney@lsw1-f1-eqiad> show firewall log | match 1.1.1.1 
   13:22:28  pfe       R      et-0/0/48.100       ICMP            1.1.1.1                          208.80.154.226
   13:22:27  pfe       R      et-0/0/48.100       ICMP            1.1.1.1                          208.80.154.226
   13:22:26  pfe       R      et-0/0/48.100       ICMP            1.1.1.1                          208.80.154.226
   13:22:25  pfe       R      et-0/0/48.100       ICMP            1.1.1.1                          208.80.154.226
   13:22:24  pfe       R      et-0/0/48.100       ICMP            1.1.1.1                          208.80.154.226

IPv6

Relevant Config

Filter applied on both spine switches:

   cmooney@lsw1-e1-eqiad> show configuration firewall family inet6 filter BLOCK_MS_FE1012_6 
   term BLOCK {
       from {
           destination-address {
               2620:0:861:ed1a::3/128;
           }
       }
       then {
           log;
           reject;
       }
   }
   term ALLOW {
       then accept;
   }
   cmooney@lsw1-f1-eqiad> show configuration firewall family inet6 filter BLOCK_MS_FE1012_6 
   term BLOCK {
       from {
           destination-address {
               2620:0:861:ed1a::3/128;
           }
       }
       then {
           log;
           reject;
       }
   }
   term ALLOW {
       then accept;
   }

Tests

Similar to the v4 test the filter is applied on the Spine device links towards the Eqiad CRs (to block return internet traffic):

   cmooney@lsw1-e1-eqiad# show | compare   
   [edit interfaces et-0/0/48 unit 100 family inet6]
   +       filter {
   +           input BLOCK_MS_FE1012_6;
   +       }
   
   
   cmooney@lsw1-f1-eqiad# show | compare 
   [edit interfaces et-0/0/48 unit 100 family inet6]
   +       filter {
   +           input BLOCK_MS_FE1012_6;
   +       }


Firstly we kick off a ping, and then commit the above config on both devices:

   root@ms-fe1012-test:~# ping -6 -I 2620:0:861:ed1a::3 one.one.one.one 
   PING one.one.one.one(one.one.one.one (2606:4700:4700::1111)) from 2620:0:861:ed1a::3 : 56 data bytes
   64 bytes from one.one.one.one (2606:4700:4700::1111): icmp_seq=1 ttl=59 time=0.390 ms
   64 bytes from one.one.one.one (2606:4700:4700::1111): icmp_seq=2 ttl=59 time=0.457 ms
   64 bytes from one.one.one.one (2606:4700:4700::1111): icmp_seq=3 ttl=59 time=0.471 ms
   64 bytes from one.one.one.one (2606:4700:4700::1111): icmp_seq=4 ttl=59 time=0.374 ms
   64 bytes from one.one.one.one (2606:4700:4700::1111): icmp_seq=5 ttl=59 time=0.422 ms
   ^C
   --- one.one.one.one ping statistics ---
   10 packets transmitted, 5 received, 50% packet loss, time 9154ms
   rtt min/avg/max/mdev = 0.374/0.422/0.471/0.037 ms


The logs on the Spine device show the packets dropped as expected:

   cmooney@lsw1-f1-eqiad> show firewall log | match 2620:0:861:ed1a::3 
   13:38:24  pfe       R      et-0/0/48.100       ICMPv6          2606:4700:4700::1111             2620:0:861:ed1a::3
   13:38:23  pfe       R      et-0/0/48.100       ICMPv6          2606:4700:4700::1111             2620:0:861:ed1a::3
   13:38:22  pfe       R      et-0/0/48.100       ICMPv6          2606:4700:4700::1111             2620:0:861:ed1a::3
   13:38:21  pfe       R      et-0/0/48.100       ICMPv6          2606:4700:4700::1111             2620:0:861:ed1a::3
   13:38:20  pfe       R      et-0/0/48.100       ICMPv6          2606:4700:4700::1111             2620:0:861:ed1a::3

IP Filters on IRB interface

This test is to validate that IP packets sourced from end-hosts connected to a Vlan by access or trunk links, and sending IP packets outside their subnet via an IRB gateway on the top-of-rack, is properly filtered by ACLs/firewall filters applied to the IRB interface.

IPv4

Relevant Configuration

New firewall filter added as follows:

   cmooney@lsw1-e3-eqiad> show configuration firewall family inet filter BLOCK_ELASTIC1093 
   term BLOCK {
       from {
           source-address {
               10.64.132.2/32;
           }
       }
       then {
           log;
           reject;
       }
   }
   term ALLOW {
       then accept;
   }

Tests

The firewall was applied to irb.1033 after a ping was initiated from elastic1093-test

   cmooney@lsw1-e3-eqiad# show | compare 
   [edit interfaces irb unit 1033 family inet]
   +       filter {
   +           input BLOCK_ELASTIC1093;
   +       }
   
   {master:0}[edit]
   cmooney@lsw1-e3-eqiad# commit 
   configuration check succeeds
   commit complete

When the filter was applied the ping stopped and host began to receive ICMP "packet filtered" messages (type 3, code 8) from the switch instead:

   root@elastic1093-test:~# ping -I 10.64.132.2 10.3.0.1
   PING 10.3.0.1 (10.3.0.1) from 10.64.132.2 : 56(84) bytes of data.
   64 bytes from 10.3.0.1: icmp_seq=1 ttl=61 time=0.234 ms
   64 bytes from 10.3.0.1: icmp_seq=2 ttl=61 time=0.246 ms
   64 bytes from 10.3.0.1: icmp_seq=3 ttl=61 time=0.264 ms
   64 bytes from 10.3.0.1: icmp_seq=4 ttl=61 time=0.189 ms
   64 bytes from 10.3.0.1: icmp_seq=5 ttl=61 time=0.257 ms
   From 10.64.132.1 icmp_seq=6 Packet filtered
   From 10.64.132.1 icmp_seq=7 Packet filtered
   From 10.64.132.1 icmp_seq=8 Packet filtered
   From 10.64.132.1 icmp_seq=9 Packet filtered

Switch firewall logs show the dropped packets too:

   cmooney@lsw1-e3-eqiad> show firewall log 
   Log :
   Time      Filter    Action Interface           Protocol        Src Addr                         Dest Addr
   10:48:26  pfe       R      xe-0/0/21.0         TCP             10.64.132.2                      10.64.132.1
   10:48:22  pfe       R      xe-0/0/21.0         TCP             10.64.132.2                      10.64.132.1
   10:48:21  pfe       R      xe-0/0/21.0         TCP             10.64.132.2                      10.64.132.1
   10:48:20  pfe       R      xe-0/0/21.0         TCP             10.64.132.2                      10.64.132.1
   10:48:20  pfe       R      xe-0/0/21.0         TCP             10.64.132.2                      10.64.132.1
   10:48:19  pfe       R      xe-0/0/21.0         TCP             10.64.132.2                      10.64.132.1
   10:48:18  pfe       R      xe-0/0/21.0         ICMP            10.64.132.2                      10.3.0.1
   10:48:17  pfe       R      xe-0/0/21.0         ICMP            10.64.132.2                      10.3.0.1
   10:48:17  pfe       R      xe-0/0/21.0         TCP             10.64.132.2                      10.64.132.1

IPv6

Same sort of test is repeated with IPv6. Test filter:

   cmooney@lsw1-e3-eqiad# run show configuration firewall family inet6 filter BLOCK-ELASTIC1093_6 
   term BLOCK {
       from {
           source-address {
               2620:0:861:10b:10:64:132:2/128;
           }
       }
       then {
           log;
           reject;
       }
   }
   term ALLOW {
       then accept;
   }

Results

Change commited as follows after ping started:

   cmooney@lsw1-e3-eqiad# show | compare 
   [edit interfaces irb unit 1033 family inet6]
   +    filter {
   +        input BLOCK-ELASTIC1093_6;
   +    }
   
   {master:0}[edit interfaces irb unit 1033]
   cmooney@lsw1-e3-eqiad# commit
   configuration check succeeds
   commit complete

Pings stopped on commit as expected, host starts getting ICMP "admin prohibited" (type 1 code 1) back from switch:

   root@elastic1093-test:~# ping -I 2620:0:861:10b:10:64:132:2 authdns1001.wikimedia.org
   PING authdns1001.wikimedia.org(authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134)) from 2620:0:861:10b:10:64:132:2 : 56 data bytes
   64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=1 ttl=61 time=0.161 ms
   64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=2 ttl=61 time=0.168 ms
   64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=3 ttl=61 time=0.176 ms
   64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=4 ttl=61 time=0.211 ms
   64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=5 ttl=61 time=0.234 ms
   64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=6 ttl=61 time=0.180 ms
   From irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1) icmp_seq=7 Destination unreachable: Administratively prohibited
   From irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1) icmp_seq=8 Destination unreachable: Administratively prohibited
   From irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1) icmp_seq=9 Destination unreachable: Administratively prohibited

Firewall logs also show the drops:

   cmooney@lsw1-e3-eqiad> show firewall log               
   Log :
   Time      Filter    Action Interface           Protocol        Src Addr                         Dest Addr
   11:54:14  pfe       R      xe-0/0/21.0         ICMPv6          2620:0:861:10b:10:64:132:2       fe80::a6e1:1a04:981:d580
   11:52:44  pfe       R      xe-0/0/21.0         ICMPv6          2620:0:861:10b:10:64:132:2       2620:0:861:2:208:80:154:134
   11:52:43  pfe       R      xe-0/0/21.0         ICMPv6          2620:0:861:10b:10:64:132:2       2620:0:861:2:208:80:154:134
   11:52:42  pfe       R      xe-0/0/21.0         ICMPv6          2620:0:861:10b:10:64:132:2       2620:0:861:2:208:80:154:134
   11:52:41  pfe       R      xe-0/0/21.0         ICMPv6          2620:0:861:10b:10:64:132:2       2620:0:861:2:208:80:154:134
   11:52:40  pfe       R      xe-0/0/21.0         ICMPv6          2620:0:861:10b:10:64:132:2       2620:0:861:2:208:80:154:134


Failover Tests

The previous set of tests validated the configuration and operation of the network in normal circumstances. We also need to verify that failover works as expected with the given configuration.

Spine Switch Failure

The first case we need to consider it what happens if a Spine switch fails. In our topology the Spine's provide all access to the other datacenter rows, as well as the outside internet. So we need to know that if one of them fails unexpectedly traffic will continue to flow via the other.

Test Setup

Eqiad Spine Failure Topo.png

A ping to an internet host was run from elastic1093-test, which is connected to Leaf switch LSW1-E3. The destination IP for the ping was one which caused LSW-E3 to ECMP it via LSW-E1, and take the path out to the internet via CR1-EQIAD. While this ping was ongoing, the power cables were removed from LSW1-E1, making the path the ping traffic was taking invalid.

Firstly we verify the traffic is routing out via E1 for the given src + dst IP:

   root@elastic1093-test:~# mtr --address 208.80.154.229 -z -b -w -c 5 1.1.1.1
   Start: 2022-02-18T07:59:07+0000
   HOST: elastic1093-test                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS???    irb-1033.lsw1-e3-eqiad.eqiad.wmnet (10.64.132.1)   0.0%     5    1.4   4.8   1.4   8.4   2.7
     2. AS???    irb-1031.lsw1-e1-eqiad.eqiad.wmnet (10.64.130.1)   0.0%     5    6.1   8.8   3.3  24.7   9.0
     3. AS???    et-1-0-2-100.cr1-eqiad.eqiad.wmnet (10.66.0.8)     0.0%     5    0.5   0.5   0.4   0.7   0.1
     4. AS14907  ae0.cr2-eqiad.wikimedia.org (208.80.154.194)       0.0%     5    0.3   0.8   0.2   2.7   1.1
     5. AS???    13335.ash.equinix.com (206.126.237.30)            60.0%     5    1.3   8.1   1.3  14.8   9.6
     6. AS13335  172.70.172.2                                       0.0%     5    7.5   3.0   1.6   7.5   2.6
     7. AS13335  one.one.one.one (1.1.1.1)                          0.0%     5    0.7   0.7   0.7   0.7   0.0

Then we kick off our ping, asking on-site staff to power down E1 once it's running:

   root@elastic1093-test:~# ping 1.1.1.1
   PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
   64 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=0.574 ms
   64 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=0.605 ms
   64 bytes from 1.1.1.1: icmp_seq=3 ttl=59 time=0.625 ms
   64 bytes from 1.1.1.1: icmp_seq=4 ttl=59 time=0.629 ms
   64 bytes from 1.1.1.1: icmp_seq=5 ttl=59 time=0.610 ms
   64 bytes from 1.1.1.1: icmp_seq=6 ttl=59 time=0.630 ms
   64 bytes from 1.1.1.1: icmp_seq=7 ttl=59 time=0.523 ms
   64 bytes from 1.1.1.1: icmp_seq=8 ttl=59 time=0.564 ms
   64 bytes from 1.1.1.1: icmp_seq=9 ttl=59 time=0.642 ms
   64 bytes from 1.1.1.1: icmp_seq=10 ttl=59 time=0.635 ms
   64 bytes from 1.1.1.1: icmp_seq=11 ttl=59 time=0.581 ms
   64 bytes from 1.1.1.1: icmp_seq=12 ttl=59 time=0.544 ms
   64 bytes from 1.1.1.1: icmp_seq=13 ttl=59 time=0.634 ms
   64 bytes from 1.1.1.1: icmp_seq=14 ttl=59 time=0.581 ms
   64 bytes from 1.1.1.1: icmp_seq=15 ttl=59 time=0.586 ms
   64 bytes from 1.1.1.1: icmp_seq=16 ttl=59 time=0.558 ms
   64 bytes from 1.1.1.1: icmp_seq=17 ttl=59 time=0.609 ms
   64 bytes from 1.1.1.1: icmp_seq=18 ttl=59 time=0.557 ms
   64 bytes from 1.1.1.1: icmp_seq=19 ttl=59 time=0.624 ms
   64 bytes from 1.1.1.1: icmp_seq=20 ttl=59 time=0.565 ms
   64 bytes from 1.1.1.1: icmp_seq=21 ttl=59 time=0.536 ms
   64 bytes from 1.1.1.1: icmp_seq=22 ttl=59 time=0.596 ms
   64 bytes from 1.1.1.1: icmp_seq=23 ttl=59 time=0.589 ms
   64 bytes from 1.1.1.1: icmp_seq=24 ttl=59 time=0.609 ms
   64 bytes from 1.1.1.1: icmp_seq=25 ttl=59 time=0.525 ms
   64 bytes from 1.1.1.1: icmp_seq=26 ttl=59 time=0.626 ms
   64 bytes from 1.1.1.1: icmp_seq=27 ttl=59 time=0.584 ms
   64 bytes from 1.1.1.1: icmp_seq=28 ttl=59 time=0.541 ms
   64 bytes from 1.1.1.1: icmp_seq=29 ttl=59 time=0.722 ms
   64 bytes from 1.1.1.1: icmp_seq=30 ttl=59 time=0.628 ms
   64 bytes from 1.1.1.1: icmp_seq=31 ttl=59 time=0.572 ms
   64 bytes from 1.1.1.1: icmp_seq=32 ttl=59 time=0.512 ms
   64 bytes from 1.1.1.1: icmp_seq=33 ttl=59 time=0.633 ms
   64 bytes from 1.1.1.1: icmp_seq=34 ttl=59 time=0.611 ms
   64 bytes from 1.1.1.1: icmp_seq=35 ttl=59 time=0.618 ms
   64 bytes from 1.1.1.1: icmp_seq=36 ttl=59 time=0.579 ms
   64 bytes from 1.1.1.1: icmp_seq=37 ttl=59 time=0.716 ms
   64 bytes from 1.1.1.1: icmp_seq=38 ttl=59 time=0.579 ms
   64 bytes from 1.1.1.1: icmp_seq=39 ttl=59 time=0.561 ms
   64 bytes from 1.1.1.1: icmp_seq=40 ttl=59 time=0.537 ms
   64 bytes from 1.1.1.1: icmp_seq=41 ttl=59 time=0.645 ms
   64 bytes from 1.1.1.1: icmp_seq=42 ttl=59 time=0.521 ms
   64 bytes from 1.1.1.1: icmp_seq=43 ttl=59 time=0.559 ms
   64 bytes from 1.1.1.1: icmp_seq=44 ttl=59 time=0.602 ms
   64 bytes from 1.1.1.1: icmp_seq=45 ttl=59 time=0.526 ms
   64 bytes from 1.1.1.1: icmp_seq=46 ttl=59 time=0.578 ms
   64 bytes from 1.1.1.1: icmp_seq=47 ttl=59 time=0.977 ms
   64 bytes from 1.1.1.1: icmp_seq=48 ttl=59 time=0.481 ms
   64 bytes from 1.1.1.1: icmp_seq=49 ttl=59 time=0.602 ms
   64 bytes from 1.1.1.1: icmp_seq=50 ttl=59 time=0.553 ms
   64 bytes from 1.1.1.1: icmp_seq=51 ttl=59 time=0.708 ms
   64 bytes from 1.1.1.1: icmp_seq=52 ttl=59 time=0.494 ms
   64 bytes from 1.1.1.1: icmp_seq=53 ttl=59 time=0.556 ms
   64 bytes from 1.1.1.1: icmp_seq=54 ttl=59 time=0.513 ms
   64 bytes from 1.1.1.1: icmp_seq=55 ttl=59 time=0.544 ms
   64 bytes from 1.1.1.1: icmp_seq=56 ttl=59 time=0.487 ms
   64 bytes from 1.1.1.1: icmp_seq=57 ttl=59 time=0.481 ms
   64 bytes from 1.1.1.1: icmp_seq=58 ttl=59 time=0.467 ms
   64 bytes from 1.1.1.1: icmp_seq=59 ttl=59 time=0.497 ms
   64 bytes from 1.1.1.1: icmp_seq=60 ttl=59 time=0.490 ms
   64 bytes from 1.1.1.1: icmp_seq=61 ttl=59 time=0.564 ms
   64 bytes from 1.1.1.1: icmp_seq=62 ttl=59 time=0.505 ms
   64 bytes from 1.1.1.1: icmp_seq=63 ttl=59 time=0.529 ms
   64 bytes from 1.1.1.1: icmp_seq=64 ttl=59 time=0.527 ms
   64 bytes from 1.1.1.1: icmp_seq=65 ttl=59 time=0.556 ms
   64 bytes from 1.1.1.1: icmp_seq=66 ttl=59 time=0.592 ms
   64 bytes from 1.1.1.1: icmp_seq=67 ttl=59 time=0.541 ms
   64 bytes from 1.1.1.1: icmp_seq=68 ttl=59 time=0.561 ms
   64 bytes from 1.1.1.1: icmp_seq=69 ttl=59 time=0.475 ms
   64 bytes from 1.1.1.1: icmp_seq=70 ttl=59 time=0.586 ms
   64 bytes from 1.1.1.1: icmp_seq=71 ttl=59 time=0.539 ms
   64 bytes from 1.1.1.1: icmp_seq=72 ttl=59 time=0.525 ms
   64 bytes from 1.1.1.1: icmp_seq=73 ttl=59 time=0.635 ms
   64 bytes from 1.1.1.1: icmp_seq=74 ttl=59 time=0.541 ms
   64 bytes from 1.1.1.1: icmp_seq=75 ttl=59 time=0.538 ms
   64 bytes from 1.1.1.1: icmp_seq=76 ttl=59 time=0.568 ms
   64 bytes from 1.1.1.1: icmp_seq=77 ttl=59 time=0.553 ms
   ^C
   --- 1.1.1.1 ping statistics ---
   77 packets transmitted, 77 received, 0% packet loss, time 77791ms
   rtt min/avg/max/mdev = 0.467/0.576/0.977/0.070 ms


Stopping the ping a short time after the confirmation LSW1-E1 was powered off we see that we have 0 pings lost, a good sign. We can also see in a traceroute that the traffic is now flowing out via LSW1-F1 to CR2-EQIAD:

   root@elastic1093-test:~# mtr --address 208.80.154.229 -z -b -w -c 5 1.1.1.1
   Start: 2022-02-18T08:08:11+0000
   HOST: elastic1093-test                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS???    irb-1033.lsw1-e3-eqiad.eqiad.wmnet (10.64.132.1)   0.0%     5    7.4   3.8   1.4   7.4   2.4
     2. AS???    irb-1035.lsw1-f1-eqiad.eqiad.wmnet (10.64.134.1)   0.0%     5    8.3   5.3   3.2   8.3   2.0
     3. AS???    et-1-0-2-100.cr2-eqiad.eqiad.wmnet (10.66.0.10)    0.0%     5    0.6   0.8   0.4   1.5   0.5
     4. AS???    13335.ash.equinix.com (206.126.237.30)            40.0%     5    1.6   7.7   1.6  19.5  10.2
     5. AS13335  172.70.172.2                                       0.0%     5    2.7  11.2   1.2  45.3  19.1
     6. AS13335  one.one.one.one (1.1.1.1)                          0.0%     5    0.7   1.0   0.7   2.4   0.8

So all looks good. The other thing we want to do is verify things return to normal after power is restored, so we kick off another ping and ask on-site to restore power to LSW1-E1:

   root@elastic1093-test:~# ping 1.1.1.1
   PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
   64 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=0.504 ms
   64 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=0.660 ms
   64 bytes from 1.1.1.1: icmp_seq=3 ttl=59 time=0.524 ms
   64 bytes from 1.1.1.1: icmp_seq=4 ttl=59 time=0.607 ms
   64 bytes from 1.1.1.1: icmp_seq=5 ttl=59 time=0.553 ms
   64 bytes from 1.1.1.1: icmp_seq=6 ttl=59 time=0.543 ms
   64 bytes from 1.1.1.1: icmp_seq=7 ttl=59 time=0.566 ms
   64 bytes from 1.1.1.1: icmp_seq=8 ttl=59 time=0.565 ms
   64 bytes from 1.1.1.1: icmp_seq=9 ttl=59 time=0.559 ms
   64 bytes from 1.1.1.1: icmp_seq=10 ttl=59 time=0.593 ms
   64 bytes from 1.1.1.1: icmp_seq=11 ttl=59 time=0.654 ms
   64 bytes from 1.1.1.1: icmp_seq=12 ttl=59 time=0.565 ms
   64 bytes from 1.1.1.1: icmp_seq=13 ttl=59 time=0.671 ms
   64 bytes from 1.1.1.1: icmp_seq=14 ttl=59 time=0.576 ms
   64 bytes from 1.1.1.1: icmp_seq=15 ttl=59 time=0.605 ms
   64 bytes from 1.1.1.1: icmp_seq=16 ttl=59 time=0.565 ms
   64 bytes from 1.1.1.1: icmp_seq=17 ttl=59 time=0.575 ms
   64 bytes from 1.1.1.1: icmp_seq=18 ttl=59 time=0.700 ms
   64 bytes from 1.1.1.1: icmp_seq=19 ttl=59 time=0.573 ms
   64 bytes from 1.1.1.1: icmp_seq=20 ttl=59 time=0.632 ms
   64 bytes from 1.1.1.1: icmp_seq=21 ttl=59 time=0.580 ms
   64 bytes from 1.1.1.1: icmp_seq=22 ttl=59 time=0.609 ms
   64 bytes from 1.1.1.1: icmp_seq=23 ttl=59 time=0.649 ms
   64 bytes from 1.1.1.1: icmp_seq=24 ttl=59 time=0.639 ms
   64 bytes from 1.1.1.1: icmp_seq=25 ttl=59 time=0.579 ms
   64 bytes from 1.1.1.1: icmp_seq=26 ttl=59 time=0.618 ms
   64 bytes from 1.1.1.1: icmp_seq=27 ttl=59 time=0.597 ms
   64 bytes from 1.1.1.1: icmp_seq=28 ttl=59 time=0.639 ms
   64 bytes from 1.1.1.1: icmp_seq=29 ttl=59 time=0.706 ms
   64 bytes from 1.1.1.1: icmp_seq=30 ttl=59 time=0.557 ms
   64 bytes from 1.1.1.1: icmp_seq=31 ttl=59 time=0.547 ms
   64 bytes from 1.1.1.1: icmp_seq=32 ttl=59 time=0.595 ms
   64 bytes from 1.1.1.1: icmp_seq=33 ttl=59 time=0.583 ms
   64 bytes from 1.1.1.1: icmp_seq=34 ttl=59 time=0.611 ms
   64 bytes from 1.1.1.1: icmp_seq=35 ttl=59 time=0.596 ms
   64 bytes from 1.1.1.1: icmp_seq=36 ttl=59 time=0.591 ms
   64 bytes from 1.1.1.1: icmp_seq=37 ttl=59 time=0.624 ms
   64 bytes from 1.1.1.1: icmp_seq=38 ttl=59 time=0.554 ms
   64 bytes from 1.1.1.1: icmp_seq=39 ttl=59 time=0.606 ms
   64 bytes from 1.1.1.1: icmp_seq=40 ttl=59 time=0.588 ms
   64 bytes from 1.1.1.1: icmp_seq=41 ttl=59 time=0.620 ms
   64 bytes from 1.1.1.1: icmp_seq=42 ttl=59 time=0.562 ms
   64 bytes from 1.1.1.1: icmp_seq=43 ttl=59 time=0.602 ms
   64 bytes from 1.1.1.1: icmp_seq=44 ttl=59 time=0.592 ms
   64 bytes from 1.1.1.1: icmp_seq=45 ttl=59 time=0.615 ms
   64 bytes from 1.1.1.1: icmp_seq=46 ttl=59 time=0.606 ms
   64 bytes from 1.1.1.1: icmp_seq=47 ttl=59 time=0.550 ms
   64 bytes from 1.1.1.1: icmp_seq=48 ttl=59 time=0.593 ms
   64 bytes from 1.1.1.1: icmp_seq=49 ttl=59 time=0.621 ms
   64 bytes from 1.1.1.1: icmp_seq=50 ttl=59 time=0.560 ms
   64 bytes from 1.1.1.1: icmp_seq=51 ttl=59 time=0.597 ms
   64 bytes from 1.1.1.1: icmp_seq=52 ttl=59 time=0.626 ms
   64 bytes from 1.1.1.1: icmp_seq=53 ttl=59 time=0.577 ms
   64 bytes from 1.1.1.1: icmp_seq=54 ttl=59 time=0.558 ms
   64 bytes from 1.1.1.1: icmp_seq=55 ttl=59 time=0.588 ms
   64 bytes from 1.1.1.1: icmp_seq=56 ttl=59 time=0.544 ms
   64 bytes from 1.1.1.1: icmp_seq=57 ttl=59 time=0.564 ms
   64 bytes from 1.1.1.1: icmp_seq=58 ttl=59 time=0.643 ms
   64 bytes from 1.1.1.1: icmp_seq=59 ttl=59 time=0.569 ms
   64 bytes from 1.1.1.1: icmp_seq=60 ttl=59 time=0.573 ms
   64 bytes from 1.1.1.1: icmp_seq=61 ttl=59 time=0.572 ms
   64 bytes from 1.1.1.1: icmp_seq=62 ttl=59 time=0.599 ms
   64 bytes from 1.1.1.1: icmp_seq=63 ttl=59 time=0.596 ms
   64 bytes from 1.1.1.1: icmp_seq=64 ttl=59 time=0.580 ms
   64 bytes from 1.1.1.1: icmp_seq=65 ttl=59 time=0.643 ms
   64 bytes from 1.1.1.1: icmp_seq=66 ttl=59 time=0.601 ms
   64 bytes from 1.1.1.1: icmp_seq=67 ttl=59 time=0.549 ms
   64 bytes from 1.1.1.1: icmp_seq=68 ttl=59 time=0.701 ms
   64 bytes from 1.1.1.1: icmp_seq=69 ttl=59 time=0.653 ms
   64 bytes from 1.1.1.1: icmp_seq=70 ttl=59 time=0.631 ms
   64 bytes from 1.1.1.1: icmp_seq=71 ttl=59 time=0.632 ms
   64 bytes from 1.1.1.1: icmp_seq=72 ttl=59 time=0.617 ms
   64 bytes from 1.1.1.1: icmp_seq=73 ttl=59 time=0.601 ms
   64 bytes from 1.1.1.1: icmp_seq=74 ttl=59 time=0.597 ms
   64 bytes from 1.1.1.1: icmp_seq=75 ttl=59 time=0.668 ms
   64 bytes from 1.1.1.1: icmp_seq=76 ttl=59 time=0.572 ms
   64 bytes from 1.1.1.1: icmp_seq=77 ttl=59 time=0.601 ms
   64 bytes from 1.1.1.1: icmp_seq=78 ttl=59 time=0.637 ms
   64 bytes from 1.1.1.1: icmp_seq=79 ttl=59 time=0.589 ms
   64 bytes from 1.1.1.1: icmp_seq=80 ttl=59 time=0.571 ms
   64 bytes from 1.1.1.1: icmp_seq=81 ttl=59 time=0.608 ms
   64 bytes from 1.1.1.1: icmp_seq=82 ttl=59 time=0.674 ms
   64 bytes from 1.1.1.1: icmp_seq=83 ttl=59 time=0.583 ms
   64 bytes from 1.1.1.1: icmp_seq=84 ttl=59 time=0.629 ms
   64 bytes from 1.1.1.1: icmp_seq=85 ttl=59 time=0.629 ms
   64 bytes from 1.1.1.1: icmp_seq=86 ttl=59 time=0.592 ms
   64 bytes from 1.1.1.1: icmp_seq=87 ttl=59 time=0.610 ms
   64 bytes from 1.1.1.1: icmp_seq=88 ttl=59 time=0.542 ms
   ^C
   --- 1.1.1.1 ping statistics ---
   88 packets transmitted, 88 received, 0% packet loss, time 89071ms
   rtt min/avg/max/mdev = 0.504/0.598/0.706/0.039 ms

Again no packets lost, and when we do a trace out we can see the traffic is flowing via lsw1-e1 again:

   root@elastic1093-test:~# mtr --address 208.80.154.229 -z -b -w -c 5 1.1.1.1
   Start: 2022-02-18T08:15:14+0000
   HOST: elastic1093-test                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS???    irb-1033.lsw1-e3-eqiad.eqiad.wmnet (10.64.132.1)   0.0%     5    1.6   5.1   1.6   8.1   3.2
     2. AS???    irb-1031.lsw1-e1-eqiad.eqiad.wmnet (10.64.130.1)   0.0%     5    6.1  13.9   0.8  51.9  21.6
     3. AS???    et-1-0-2-100.cr1-eqiad.eqiad.wmnet (10.66.0.8)     0.0%     5    0.5   0.5   0.4   0.6   0.1
     4. AS14907  ae0.cr2-eqiad.wikimedia.org (208.80.154.194)       0.0%     5    0.3   0.6   0.3   1.6   0.6
     5. AS???    ???                                               100.0     5    0.0   0.0   0.0   0.0   0.0
     6. AS13335  172.70.172.2                                       0.0%     5    2.2   1.8   1.2   2.2   0.4
     7. AS13335  one.one.one.one (1.1.1.1)                          0.0%     5    0.7   0.7   0.7   0.7   0.0

Leaf to Spine Link Failure

This was not tested separately as the 'Spine Link Failure' effectively, from the Leaf switch's point of view, has the same effect as the link to the Spine going down. It has no way to tell if it's a device failure or just the link, so it will behave in the same way. As such there is no need to run a separate test.

Spine to CR Link Failure

Eqiad Spine CR Link Failure.png

In this test we want to pull the cable running from lsw1-e1 to cr1-eqiad, and observe what happens. Testing is again run from elastic1093-test, connected to lsw1-e3. LSW1-E3 will not notice any physical change or difference in IGP topology, however it should get an EVPN BGP WITHDRAW for the default route it was receiving from LSW1-E1, with it's own VTEP IP as next-hop. LSW1-E3 will still receive a default BGP route from LSW1-E1, however this will be the route it knows from LSW1-F1, and the next-hop VTEP will be that of the other switch.

This process should happen very quickly, and all traffic flow from LSW1-E3 to LSW1-F1 shortly after the cable pull and resulting BGP WITHDRAW.

Results

Firstly we establish that our ping routes out from LSW1-E3 via LSW1-E1 for our chosen src and destination:

   root@elastic1093-test:~# mtr --address 2620:0:861:ed1a::6 -z -b -w -c 5 www.ietf.org
   Start: 2022-02-18T08:15:56+0000
   HOST: elastic1093-test                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS14907  irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1)    0.0%     5    0.5   0.5   0.5   0.6   0.0
     2. AS14907  irb-1039.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:100::1)    0.0%     5    0.5   0.5   0.5   0.6   0.0
     3. AS14907  et-1-0-2-100.cr1-eqiad.eqiad.wmnet (2620:0:861:fe07::1)   0.0%     5    0.8   0.6   0.4   0.8   0.2
     4. AS14907  ae0.cr2-eqiad.wikimedia.org (2620:0:861:fe00::2)          0.0%     5    2.1   0.9   0.4   2.1   0.7
     5. AS???    13335.ash.equinix.com (2001:504:0:2:0:1:3335:1)          80.0%     5    2.1   2.1   2.1   2.1   0.0
     6. AS13335  2400:cb00:354:3::                                         0.0%     5    0.4   3.9   0.4   8.2   3.5
     7. AS13335  2606:4700::6810:2d63                                      0.0%     5    0.5   1.3   0.4   4.6   1.8

Next we kick off the ping, and ask DC-Ops to pull the cable in the meantime:

   root@elastic1093-test:~# ping6 -I 2620:0:861:ed1a::6 www.ietf.org
   PING www.ietf.org(2606:4700::6810:2d63 (2606:4700::6810:2d63)) from 2620:0:861:ed1a::6 : 56 data bytes
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=1 ttl=59 time=4.69 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=2 ttl=59 time=0.440 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=3 ttl=59 time=0.364 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=4 ttl=59 time=0.434 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=5 ttl=59 time=0.374 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=6 ttl=59 time=0.470 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=7 ttl=59 time=0.454 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=8 ttl=59 time=0.489 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=9 ttl=59 time=0.376 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=10 ttl=59 time=0.709 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=11 ttl=59 time=4.29 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=12 ttl=59 time=1.28 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=13 ttl=59 time=0.458 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=14 ttl=59 time=0.414 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=15 ttl=59 time=14.4 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=16 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=17 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=18 ttl=59 time=0.463 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=19 ttl=59 time=0.446 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=20 ttl=59 time=0.457 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=21 ttl=59 time=0.446 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=22 ttl=59 time=0.432 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=23 ttl=59 time=0.458 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=24 ttl=59 time=0.440 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=25 ttl=59 time=0.426 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=26 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=27 ttl=59 time=0.434 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=28 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=29 ttl=59 time=0.340 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=30 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=31 ttl=59 time=0.461 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=32 ttl=59 time=0.432 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=33 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=34 ttl=59 time=0.427 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=35 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=36 ttl=59 time=0.425 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=37 ttl=59 time=0.346 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=38 ttl=59 time=0.435 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=39 ttl=59 time=0.425 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=40 ttl=59 time=0.439 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=41 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=42 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=43 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=44 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=45 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=46 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=47 ttl=59 time=0.425 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=48 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=49 ttl=59 time=0.427 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=50 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=51 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=52 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=53 ttl=59 time=0.446 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=54 ttl=59 time=0.351 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=55 ttl=59 time=0.437 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=56 ttl=59 time=0.429 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=57 ttl=59 time=0.452 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=58 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=59 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=60 ttl=59 time=0.457 ms
   ^C
   --- www.ietf.org ping statistics ---
   60 packets transmitted, 60 received, 0% packet loss, time 59427ms
   rtt min/avg/max/mdev = 0.340/0.814/14.417/1.917 ms


Zero packet's lost again which is good, let's verify the traffic is taking the new path:

   root@elastic1093-test:~# mtr --address 2620:0:861:ed1a::6 -z -b -w -c 5 www.ietf.org
   Start: 2022-02-18T08:21:40+0000
   HOST: elastic1093-test                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS14907  irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1)    0.0%     5    0.5   0.5   0.5   0.6   0.0
     2. AS14907  irb-1035.lsw1-f1-eqiad.eqiad.wmnet (2620:0:861:10d::1)    0.0%     5    0.7  10.2   0.6  48.3  21.3
     3. AS14907  et-1-0-2-100.cr2-eqiad.eqiad.wmnet (2620:0:861:fe08::1)   0.0%     5    0.7   0.5   0.4   0.7   0.1
     4. AS???    13335.ash.equinix.com (2001:504:0:2:0:1:3335:1)          40.0%     5   12.0   7.5   1.5  12.0   5.4
     5. AS13335  2400:cb00:350:3::                                         0.0%     5    4.7   2.0   1.1   4.7   1.5
     6. AS13335  2606:4700::6810:2c63                                      0.0%     5    0.5   0.4   0.4   0.5   0.0

All looking good so we ask on-site to restore the link, and monitor to make sure there are no issues:

   root@elastic1093-test:~# ping6 -I 2620:0:861:ed1a::6 www.ietf.org
   PING www.ietf.org(2606:4700::6810:2d63 (2606:4700::6810:2d63)) from 2620:0:861:ed1a::6 : 56 data bytes
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=1 ttl=59 time=0.370 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=2 ttl=59 time=0.353 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=3 ttl=59 time=0.434 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=4 ttl=59 time=0.442 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=5 ttl=59 time=0.428 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=6 ttl=59 time=0.438 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=7 ttl=59 time=0.341 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=8 ttl=59 time=0.335 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=9 ttl=59 time=0.361 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=10 ttl=59 time=0.425 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=11 ttl=59 time=5.31 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=12 ttl=59 time=0.353 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=13 ttl=59 time=0.448 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=14 ttl=59 time=0.420 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=15 ttl=59 time=0.447 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=16 ttl=59 time=0.425 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=17 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=18 ttl=59 time=0.335 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=19 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=20 ttl=59 time=0.427 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=21 ttl=59 time=0.606 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=22 ttl=59 time=0.475 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=23 ttl=59 time=0.444 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=24 ttl=59 time=0.425 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=25 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=26 ttl=59 time=0.444 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=27 ttl=59 time=0.338 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=28 ttl=59 time=0.342 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=29 ttl=59 time=0.426 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=30 ttl=59 time=0.331 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=31 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=32 ttl=59 time=0.425 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=33 ttl=59 time=0.463 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=34 ttl=59 time=0.496 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=35 ttl=59 time=0.538 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=36 ttl=59 time=0.420 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=37 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=38 ttl=59 time=0.794 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=39 ttl=59 time=0.608 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=40 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=41 ttl=59 time=0.438 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=42 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=43 ttl=59 time=0.545 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=44 ttl=59 time=0.542 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=45 ttl=59 time=0.426 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=46 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=47 ttl=59 time=0.483 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=48 ttl=59 time=0.421 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=49 ttl=59 time=0.446 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=50 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=51 ttl=59 time=0.434 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=52 ttl=59 time=0.440 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=53 ttl=59 time=0.448 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=54 ttl=59 time=0.638 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=55 ttl=59 time=0.452 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=56 ttl=59 time=0.475 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=57 ttl=59 time=0.466 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=58 ttl=59 time=0.401 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=59 ttl=59 time=0.522 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=60 ttl=59 time=1.22 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=61 ttl=59 time=0.441 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=62 ttl=59 time=0.448 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=63 ttl=59 time=0.426 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=64 ttl=59 time=0.340 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=65 ttl=59 time=0.441 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=66 ttl=59 time=0.547 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=67 ttl=59 time=0.362 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=68 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=69 ttl=59 time=0.373 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=70 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=71 ttl=59 time=0.441 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=72 ttl=59 time=0.520 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=73 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=74 ttl=59 time=0.641 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=75 ttl=59 time=0.432 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=76 ttl=59 time=1.78 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=77 ttl=59 time=0.440 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=78 ttl=59 time=0.428 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=79 ttl=59 time=0.421 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=80 ttl=59 time=0.430 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=81 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=82 ttl=59 time=0.380 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=83 ttl=59 time=0.451 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=84 ttl=59 time=0.440 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=85 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=86 ttl=59 time=0.387 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=87 ttl=59 time=2.15 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=88 ttl=59 time=0.473 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=89 ttl=59 time=0.523 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=90 ttl=59 time=0.444 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=91 ttl=59 time=0.421 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=92 ttl=59 time=0.452 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=93 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=94 ttl=59 time=0.457 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=95 ttl=59 time=0.351 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=96 ttl=59 time=0.518 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=97 ttl=59 time=0.551 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=98 ttl=59 time=0.439 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=99 ttl=59 time=0.440 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=100 ttl=59 time=0.446 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=101 ttl=59 time=0.445 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=102 ttl=59 time=0.501 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=103 ttl=59 time=0.455 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=104 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=105 ttl=59 time=0.443 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=106 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=107 ttl=59 time=0.444 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=108 ttl=59 time=0.457 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=109 ttl=59 time=0.443 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=110 ttl=59 time=0.451 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=111 ttl=59 time=0.442 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=112 ttl=59 time=0.335 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=113 ttl=59 time=0.421 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=114 ttl=59 time=0.452 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=115 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=116 ttl=59 time=0.718 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=117 ttl=59 time=0.377 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=118 ttl=59 time=0.416 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=119 ttl=59 time=0.444 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=120 ttl=59 time=3.95 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=121 ttl=59 time=0.529 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=122 ttl=59 time=0.447 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=123 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=124 ttl=59 time=0.444 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=125 ttl=59 time=0.435 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=126 ttl=59 time=0.466 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=127 ttl=59 time=0.451 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=128 ttl=59 time=0.440 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=129 ttl=59 time=0.435 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=130 ttl=59 time=0.457 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=131 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=132 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=133 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=134 ttl=59 time=0.460 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=135 ttl=59 time=0.336 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=136 ttl=59 time=0.460 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=137 ttl=59 time=0.449 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=138 ttl=59 time=0.420 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=139 ttl=59 time=0.421 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=140 ttl=59 time=0.465 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=141 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=142 ttl=59 time=0.435 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=143 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=144 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=145 ttl=59 time=0.462 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=146 ttl=59 time=0.443 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=147 ttl=59 time=0.449 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=148 ttl=59 time=0.442 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=149 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=150 ttl=59 time=0.353 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=151 ttl=59 time=0.463 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=152 ttl=59 time=0.428 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=153 ttl=59 time=0.466 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=154 ttl=59 time=0.441 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=155 ttl=59 time=0.455 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=156 ttl=59 time=0.453 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=157 ttl=59 time=0.422 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=158 ttl=59 time=0.461 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=159 ttl=59 time=0.387 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=160 ttl=59 time=0.434 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=161 ttl=59 time=0.449 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=162 ttl=59 time=0.441 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=163 ttl=59 time=0.364 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=164 ttl=59 time=0.431 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=165 ttl=59 time=0.428 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=166 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=167 ttl=59 time=0.433 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=168 ttl=59 time=0.465 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=169 ttl=59 time=0.439 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=170 ttl=59 time=0.456 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=171 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=172 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=173 ttl=59 time=0.449 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=174 ttl=59 time=0.499 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=175 ttl=59 time=0.449 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=176 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=177 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=178 ttl=59 time=4.20 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=179 ttl=59 time=0.495 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=180 ttl=59 time=0.357 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=181 ttl=59 time=0.433 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=182 ttl=59 time=0.441 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=183 ttl=59 time=0.446 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=184 ttl=59 time=0.354 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=185 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=186 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=187 ttl=59 time=0.434 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=188 ttl=59 time=0.424 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=189 ttl=59 time=0.425 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=190 ttl=59 time=0.481 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=191 ttl=59 time=0.459 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=192 ttl=59 time=0.445 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=193 ttl=59 time=0.423 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=194 ttl=59 time=0.535 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=195 ttl=59 time=0.440 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=196 ttl=59 time=0.456 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=197 ttl=59 time=0.363 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=198 ttl=59 time=0.421 ms
   64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=199 ttl=59 time=0.426 ms
   ^C
   --- www.ietf.org ping statistics ---
   199 packets transmitted, 199 received, 0% packet loss, time 199094ms
   rtt min/avg/max/mdev = 0.331/0.521/5.307/0.525 ms
   root@elastic1093-test:~# mtr --address 2620:0:861:ed1a::6 -z -b -w -c 5 www.ietf.org
   Start: 2022-02-18T08:25:48+0000
   HOST: elastic1093-test                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. AS14907  irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1)    0.0%     5    0.6   1.7   0.5   6.2   2.5
     2. AS14907  irb-1039.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:100::1)    0.0%     5    0.7   1.0   0.5   2.8   1.0
     3. AS14907  et-1-0-2-100.cr1-eqiad.eqiad.wmnet (2620:0:861:fe07::1)   0.0%     5    0.4   0.5   0.4   0.6   0.1
     4. AS14907  ae0.cr2-eqiad.wikimedia.org (2620:0:861:fe00::2)          0.0%     5    0.5   0.6   0.3   1.2   0.4
     5. AS???    13335.ash.equinix.com (2001:504:0:2:0:1:3335:1)          80.0%     5   29.6  29.6  29.6  29.6   0.0
     6. AS13335  2400:cb00:354:3::                                         0.0%     5    1.1   1.8   0.5   5.0   1.8
     7. AS13335  2606:4700::6810:2d63                                      0.0%     5    0.5   0.6   0.5   0.8   0.1

Again no lost packets, and we can see that by the time we are finished traffic is back on the original path.