VXLAN-EVPN Network Testing - Eqiad Expansion
This page documents various tests done to validate the correct operation of networking devices in the new cage (rows E/F) in Eqiad (Equinix, Ashburn). The switches in these rows has been set up differently to those in other rows, following an updated deisgn.
The principal change in the new design is the use of per-rack Vlans/subnets where possible, with end systems connected to an overlay network built with VXLAN/EVPN. The use of VXLAN/EVPN ensures that any edge-cases requiring layer-2 adjacency between racks can be addressed, if needed. The preference is to minimize where and when Vlans need to be extended between racks, however.
Test Topology
Tests were carried out with the above physical topology. Purple/blue links to the CR routers use 100GBase-CWDM4 and terminate into the overlay network (vrf) at layer-3 on the switches. The green links connect devices in the VXLAN fabric together, again using 100GBase-CWDM4, and provide the underlay network transport. Finally the red links are 10G DAC connections within the racks from server to switch, and terminate into overlay Vlans on the switches.
4 Dell servers, which will ultimately be the first 3 hosts provisioned in the new cage, were commandeered temporarily to perform the testing. A vanilla debian install (from debian ISO image) was used for most of the testing.
As can be seen the full Clos topology as described in the design was not ready before launch. The Juniper QFX5120-32C Spine-layer switches were unavailable due to issues with the supplier / global chip shortage. As a result of this two of the LEAF layer devices (lsw1-e1/f1, QFX5120-48Y) were configured to temporarily operate as both access and aggregation devices. These devices have enough ports to support the 6 remaining Leaf switches installed in phase 1 of the expansion. These devices, E2-E4 and F2-F4, are shown at the bottom and are also QFX5120-48Y switches.
For most tests the temporary topology does not make a difference. It does create a situation where servers connected in separate racks can have a different number of hops to reach each other, and potentially different bandwidth available, but the logical function of the network can still be assessed. Certain tests, mainly relating to failover, need to be initiated from the switches at the bottom, so elastic1093-test was used for those.
Port configurations were varied occasionally to test different features. Where relevant, device configuration snippets are included to show the setup for a test. If not explicitly detailed it can be assumed that configuration for each test has not diverged from the previous.
Functional Tests
L2 Access port
This is a very basic test to validate the normal configuration for an access port works as expected and MAC addresses are learnt by the top-of-rack switch when unicast frames are transmitted by an end host.
Relevant config
ms-fe1012-test was placed into vlan 1031 and the link brought up:
cmooney@lsw1-e1-eqiad> show configuration vlans private1-e1-eqiad
vlan-id 1031;
l3-interface irb.1031;
vxlan {
vni 2001031;
}
cmooney@lsw1-e1-eqiad> show configuration interfaces interface-range vlan-private1-e1-eqiad
member xe-0/0/7;
mtu 9192;
unit 0 {
family ethernet-switching {
interface-mode access;
vlan {
members private1-e1-eqiad;
}
}
}
cmooney@lsw1-e1-eqiad> show configuration interfaces xe-0/0/7
description "ms-fe1012 {#5004}";
root@ms-fe1012-test:~# ip -br link show enp101s0f0np0 enp101s0f0np0 UP e4:3d:1a:12:b1:00 <BROADCAST,MULTICAST,UP,LOWER_UP>
Results
MAC address learnt on port as expected:
cmooney@lsw1-e1-eqiad> show ethernet-switching table vlan-name private1-e1-eqiad
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 1 entries, 1 learned
Routing instance : default-switch
Vlan MAC MAC Logical SVLBNH/ Active
name address flags interface VENH Index source
private1-e1-eqiad e4:3d:1a:12:b1:00 D xe-0/0/7.0
L2 Trunk port
This is a similar basic test to validate the normal configuration for a trunk port works as expected.
Relevant config
an-worker1147 was connected to vlans 1031 (private1-e1-eqiad) and 1039 (analytics1-e1-eqiad) with a 802.1q trunk as follows:
cmooney@lsw1-e1-eqiad> show configuration vlans analytics1-e1-eqiad
vlan-id 1039;
l3-interface irb.1039;
vxlan {
vni 2001039;
}
cmooney@lsw1-e1-eqiad> show configuration vlans private1-e1-eqiad
vlan-id 1031;
l3-interface irb.1031;
vxlan {
vni 2001031;
}
cmooney@lsw1-e1-eqiad> show configuration interfaces xe-0/0/6
description an-worker1147;
native-vlan-id 1039;
mtu 9192;
unit 0 {
family ethernet-switching {
interface-mode trunk;
vlan {
members [ private1-e1-eqiad analytics1-e1-eqiad ];
}
}
}
root@an-worker1147-test:~# ip -d link show eno2np1
5: eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether e4:3d:1a:54:14:45 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 60 maxmtu 9600 addrgenmode eui64 numtxqueues 74 numrxqueues 74 gso_max_size 65536 gso_max_segs 65535 portname p1 switchid 441454feff1a3de4
altname enp24s0f1np1
root@an-worker1147-test:~# ip -d link show eno2np1.1031
6: eno2np1.1031@eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether e4:3d:1a:54:14:45 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65535
vlan protocol 802.1Q id 1031 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
Results
MAC addresses learnt on both Vlans as expected:
cmooney@lsw1-e1-eqiad> show ethernet-switching table interface xe-0/0/6.0 | match xe-0/0/6
MAC database for interface xe-0/0/6.0
analytics1-e1-eqiad e4:3d:1a:54:14:45 D xe-0/0/6.0
private1-e1-eqiad e4:3d:1a:54:14:45 D xe-0/0/6.0
Traffic on non-native vlan received by end host with tags:
root@an-worker1147-test:~# tcpdump -i eno2np1 -l -p -nn -e port not 22 and net 10.64.0.0/10 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eno2np1, link-type EN10MB (Ethernet), snapshot length 262144 bytes 13:08:13.276275 a4:e1:1a:81:3a:80 > e4:3d:1a:54:14:45, ethertype 802.1Q (0x8100), length 102: vlan 1031, p 0, ethertype IPv4 (0x0800), 10.64.130.1 > 10.64.130.11: ICMP echo request, id 6677, seq 0, length 64 13:08:13.276315 e4:3d:1a:54:14:45 > a4:e1:1a:81:3a:80, ethertype 802.1Q (0x8100), length 102: vlan 1031, p 0, ethertype IPv4 (0x0800), 10.64.130.11 > 10.64.130.1: ICMP echo reply, id 6677, seq 0, length 64 13:08:14.276074 a4:e1:1a:81:3a:80 > e4:3d:1a:54:14:45, ethertype 802.1Q (0x8100), length 102: vlan 1031, p 0, ethertype IPv4 (0x0800), 10.64.130.1 > 10.64.130.11: ICMP echo request, id 6677, seq 1, length 64 13:08:14.276107 e4:3d:1a:54:14:45 > a4:e1:1a:81:3a:80, ethertype 802.1Q (0x8100), length 102: vlan 1031, p 0, ethertype IPv4 (0x0800), 10.64.130.11 > 10.64.130.1: ICMP echo reply, id 6677, seq 1, length 64 13:08:36.097539 e4:3d:1a:54:14:45 > a4:e1:1a:81:3a:80, ethertype IPv4 (0x0800), length 81: 10.64.138.11.51007 > 10.3.0.1.53: 59784+ A? 1.debian.pool.ntp.org. (39) 13:08:36.097559 e4:3d:1a:54:14:45 > a4:e1:1a:81:3a:80, ethertype IPv4 (0x0800), length 81: 10.64.138.11.51007 > 10.3.0.1.53: 40077+ AAAA? 1.debian.pool.ntp.org. (39)
Routed port in VRF
Very simple test to confirm link-layer IP connectivity of a routed port terminated into an overlay VRF.
Relevant config
cmooney@lsw1-e1-eqiad> show configuration interfaces et-0/0/48 unit 100
description "cr1-eqiad et-1/0/2.100 - Production VRF";
vlan-id 100;
family inet {
address 10.66.0.9/31;
}
family inet6 {
address 2620:0:861:fe07::2/64;
}
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION | display set | match et-0/0/48 set routing-instances PRODUCTION interface et-0/0/48.100
Results
IPv4
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 10.66.0.8 size 9000 do-not-fragment count 2 PING 10.66.0.8 (10.66.0.8): 9000 data bytes 9008 bytes from 10.66.0.8: icmp_seq=0 ttl=64 time=2.722 ms 9008 bytes from 10.66.0.8: icmp_seq=1 ttl=64 time=2.896 ms --- 10.66.0.8 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 2.722/2.809/2.896/0.087 ms
IPv6
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 2620:0:861:fe07::1 size 9000 do-not-fragment count 2 source 2620:0:861:fe07::2 PING6(9048=40+8+9000 bytes) 2620:0:861:fe07::2 --> 2620:0:861:fe07::1 9008 bytes from 2620:0:861:fe07::1, icmp_seq=0 hlim=64 time=2.915 ms 9008 bytes from 2620:0:861:fe07::1, icmp_seq=1 hlim=64 time=2.719 ms --- 2620:0:861:fe07::1 ping6 statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/std-dev = 2.719/2.817/2.915/0.098 ms
Standard IP gateway / IRB in VRF
This test confirms routing via an IRB interface within an overlay routing instance works as expected.
Relevant Config
cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1031
description private1-e1-eqiad;
family inet {
address 10.64.130.1/24;
}
family inet6 {
address 2620:0:861:109::1/64;
}
root@ms-fe1012-test:~# ip -br addr show enp101s0f0np0 enp101s0f0np0 UP 10.64.130.10/24 2620:0:861:109::10/64 fe80::e63d:1aff:fe12:b100/64
Results
IPv4
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 10.64.130.0
PRODUCTION.inet.0: 34 destinations, 36 routes (34 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both
10.64.130.0/24 *[Direct/0] 5d 20:50:06
> via irb.1031
cmooney@lsw1-e1-eqiad> show arp interface irb.1031 no-resolve MAC Address Address Interface Flags e4:3d:1a:12:b1:00 10.64.130.10 irb.1031 [xe-0/0/7.0] permanent remote
root@ms-fe1012-test:~# ping -c 2 10.64.130.1 PING 10.64.130.1 (10.64.130.1) 56(84) bytes of data. 64 bytes from 10.64.130.1: icmp_seq=1 ttl=64 time=5.55 ms 64 bytes from 10.64.130.1: icmp_seq=2 ttl=64 time=6.90 ms
root@ms-fe1012-test:~# ip -4 neigh show 10.64.130.1 dev enp101s0f0np0 lladdr a4:e1:1a:81:3a:80 REACHABLE
IPv6
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 2620:0:861:fe07::
PRODUCTION.inet6.0: 37 destinations, 39 routes (37 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
2620:0:861:fe07::/64
*[Direct/0] 4d 14:45:08
> via et-0/0/48.100
cmooney@lsw1-e1-eqiad> show ipv6 neighbors interface irb.1031 IPv6 Address Linklayer Address State Exp Rtr Secure Interface 2620:0:861:109::10 e4:3d:1a:12:b1:00 reachable 0 no no irb.1031 [xe-0/0/7.0]
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 2620:0:861:109::10 source 2620:0:861:109::1 PING6(56=40+8+8 bytes) 2620:0:861:109::1 --> 2620:0:861:109::10 16 bytes from 2620:0:861:109::10, icmp_seq=0 hlim=64 time=0.633 ms 16 bytes from 2620:0:861:109::10, icmp_seq=1 hlim=64 time=0.511 ms
root@ms-fe1012-test:~# ping -c 2 2620:0:861:109::1 PING 2620:0:861:109::1(2620:0:861:109::1) 56 data bytes 64 bytes from 2620:0:861:109::1: icmp_seq=1 ttl=64 time=0.475 ms 64 bytes from 2620:0:861:109::1: icmp_seq=2 ttl=64 time=0.470 ms
root@ms-fe1012-test:~# ip -6 neigh show 2620:0:861:109::1 dev enp101s0f0np0 lladdr a4:e1:1a:81:3a:80 STALE
Remote MAC Learning on Vlan/VNI
This test validates MAC address distribution across multiple switches, using BGP EVPN. In order to perform the test Vlan private1-f1-eqiad was added temporarily to switch LSW1-E1, and trunked out to an-worker1147-test.
Relevant Config
cmooney@lsw1-e1-eqiad> show configuration interfaces xe-0/0/6
description an-worker1147;
native-vlan-id 1039;
mtu 9192;
unit 0 {
family ethernet-switching {
interface-mode trunk;
vlan {
members [ private1-e1-eqiad private1-f1-eqiad analytics1-e1-eqiad ];
}
}
}
Results
MAC address is learnt locally on lsw-e1 on port xe-0/0/6.0:
cmooney@lsw1-e1-eqiad> show ethernet-switching table vlan-name private1-f1-eqiad
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 3 entries, 3 learned
Routing instance : default-switch
Vlan MAC MAC Logical SVLBNH/ Active
name address flags interface VENH Index source
private1-f1-eqiad 00:00:5e:11:fa:ce DRP esi.1797 1796 05:00:00:fd:2a:00:1e:88:8b:00
private1-f1-eqiad a4:e1:1a:81:9e:80 DR vtep.32770 10.64.128.7
private1-f1-eqiad e4:3d:1a:54:14:45 D xe-0/0/6.0
BGP EVPN type 2 route containing this MAC address is received on lsw-f1 as a result. MAC is learnt on VNI 200135, with protocol next-hop of lsw-e1 loopback/VTEP IP.
cmooney@lsw1-f1-eqiad> show route protocol bgp table bgp.evpn.0 evpn-mac-address e4:3d:1a:54:14:45 extensive
2:10.64.128.3:64810::2001035::e4:3d:1a:54:14:45/304 MAC/IP (1 entry, 1 announced)
TSI:
Page 0 idx 0, (group EVPN_RR_CLIENTS type Internal) Type 1 val 0xe3c20a4 (adv_entry)
Advertised metrics:
Nexthop: 10.64.128.3
Localpref: 100
AS path: [64810] I
Communities: target:64810:1035 encapsulation:vxlan(0x8)
Cluster ID: 10.64.128.0
Originator ID: 10.64.128.3
Advertise: 0000003f
Path 2:10.64.128.3:64810::2001035::e4:3d:1a:54:14:45
from 10.64.128.3
Vector len 4. Val: 0
*BGP Preference: 170/-101
Route Distinguisher: 10.64.128.3:64810
Next hop type: Indirect, Next hop index: 0
Address: 0xc6864c8
Next-hop reference count: 109
Source: 10.64.128.3
Protocol next hop: 10.64.128.3
Indirect next hop: 0x2 no-forward INH Session ID: 0x0
State: <Active Int Ext>
Local AS: 64810 Peer AS: 64810
Age: 1:14:23 Metric2: 8
Validation State: unverified
Task: BGP_64810.10.64.128.3
Announcement bits (1): 1-BGP_RT_Background
AS path: I
Communities: target:64810:1035 encapsulation:vxlan(0x8)
Import Accepted
Route Label: 2001035
ESI: 00:00:00:00:00:00:00:00:00:00
Localpref: 100
Router ID: 10.64.128.3
Secondary Tables: default-switch.evpn.0
Thread: junos-main
Indirect next hops: 1
Protocol next hop: 10.64.128.3 Metric: 8
Indirect next hop: 0x2 no-forward INH Session ID: 0x0
Indirect path forwarding next hops: 1
Next hop type: Router
Next hop: 10.64.129.6 via et-0/0/52.0
Session Id: 0x0
10.64.128.3/32 Originating RIB: inet.0
Metric: 8 Node path count: 1
Forwarding nexthops: 1
Next hop type: Router
Next hop: 10.64.129.6 via et-0/0/52.0
Session Id: 0x0
The MAC is placed into the Vlan forwarding table as expected:
cmooney@lsw1-f1-eqiad> show ethernet-switching table vlan-name private1-f1-eqiad
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 4 entries, 4 learned
Routing instance : default-switch
Vlan MAC MAC Logical SVLBNH/ Active
name address flags interface VENH Index source
private1-f1-eqiad 00:00:5e:11:fa:ce DRP esi.1780 1779 05:00:00:fd:2a:00:1e:88:8b:00
private1-f1-eqiad a4:e1:1a:81:3a:80 DR vtep.32769 10.64.128.3
private1-f1-eqiad e4:3d:1a:54:14:45 DR vtep.32769 10.64.128.3
private1-f1-eqiad e4:3d:1a:54:ab:a7 D xe-0/0/6.0
Client to Client L2 Unicast forwarding
These tests verify that end hosts can communicate directly amongst themselves within a particular Vlan.
Same Switch (Pure L2)
Relevant Config
cmooney@lsw1-e1-eqiad> show configuration interfaces xe-0/0/6
description an-worker1147;
native-vlan-id 1039;
mtu 9192;
unit 0 {
family ethernet-switching {
interface-mode trunk;
vlan {
members [ private1-e1-eqiad private1-f1-eqiad analytics1-e1-eqiad ];
}
}
}
cmooney@lsw1-e1-eqiad> show configuration interfaces interface-range vlan-private1-e1-eqiad
member xe-0/0/7;
mtu 9192;
unit 0 {
family ethernet-switching {
interface-mode access;
vlan {
members private1-e1-eqiad;
}
}
}
cmooney@lsw1-e1-eqiad> show configuration interfaces xe-0/0/7
description "ms-fe1012 {#5004}";
root@ms-fe1012-test:~# ip -d link show enp101s0f0np0 2: enp101s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether e4:3d:1a:12:b1:00 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 60 maxmtu 9600 addrgenmode eui64 numtxqueues 74 numrxqueues 74 gso_max_size 65536 gso_max_segs 65535 portname p0 switchid 00b112feff1a3de4
root@an-worker1147-test:~# ip -d link show eno2np1.1031 6: eno2np1.1031@eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether e4:3d:1a:54:14:45 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65535 vlan protocol 802.1Q id 1031 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
Results
cmooney@lsw1-e1-eqiad> show ethernet-switching table vlan-name private1-e1-eqiad
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
Vlan MAC MAC Logical SVLBNH/ Active
name address flags interface VENH Index source
private1-e1-eqiad e4:3d:1a:12:b1:00 D xe-0/0/7.0
private1-e1-eqiad e4:3d:1a:54:14:45 D xe-0/0/6.0
root@an-worker1147-test:~# ping -c 2 10.64.130.10 PING 10.64.130.10 (10.64.130.10) 56(84) bytes of data. 64 bytes from 10.64.130.10: icmp_seq=1 ttl=64 time=0.172 ms 64 bytes from 10.64.130.10: icmp_seq=2 ttl=64 time=0.209 ms
root@ms-fe1012-test:~# tcpdump -i enp101s0f0np0 -l -p -nn -e icmp tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on enp101s0f0np0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 20:41:55.359338 e4:3d:1a:54:14:45 > e4:3d:1a:12:b1:00, ethertype IPv4 (0x0800), length 98: 10.64.130.11 > 10.64.130.10: ICMP echo request, id 54201, seq 20, length 64 20:41:55.359373 e4:3d:1a:12:b1:00 > e4:3d:1a:54:14:45, ethertype IPv4 (0x0800), length 98: 10.64.130.10 > 10.64.130.11: ICMP echo reply, id 54201, seq 20, length 64
Remote Switch (VXLAN tunneled)
Relevant Config
Same as for test 2.5: Remote MAC Learning on Vlan/VNI
Results
Same MAC tables as for test 2.5: Remote MAC Learning on Vlan/VNI
root@an-worker1148-test:~# ping 10.64.134.72 PING 10.64.134.72 (10.64.134.72) 56(84) bytes of data. 64 bytes from 10.64.134.72: icmp_seq=1 ttl=64 time=19.3 ms 64 bytes from 10.64.134.72: icmp_seq=2 ttl=64 time=0.180 ms
root@an-worker1147-test:~# tcpdump -i eno2np1.1035 -e -l -p -nn icmp tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eno2np1.1035, link-type EN10MB (Ethernet), snapshot length 262144 bytes 20:48:52.448479 e4:3d:1a:54:ab:a7 > e4:3d:1a:54:14:45, ethertype IPv4 (0x0800), length 98: 10.64.134.11 > 10.64.134.72: ICMP echo request, id 62059, seq 28, length 64 20:48:52.448505 e4:3d:1a:54:14:45 > e4:3d:1a:54:ab:a7, ethertype IPv4 (0x0800), length 98: 10.64.134.72 > 10.64.134.11: ICMP echo reply, id 62059, seq 28, length 64
Client to Client broadcast forwarding / ingress replication
These test verify broadcasts are correctly flooded to all ports in the Vlan, both local or remote.
Same Switch (Pure L2)
Test involves generating a broadcast on ms-fe1012-test into Vlan1031 / private-e1-eqiad, and confirm it is received on an-worker1147-test. Both of these hosts are connected to lsw-e1-eqiad, so the frame will be processed as a regular L2 frame between local ports in the same Vlan.
Transmit side:
root@ms-fe1012-test:~# echo "test broadcast in vlan 1031 private-e1-eqiad" | nc -u -b 255.255.255.255 12101
Receive side:
root@an-worker1147-test:~# tcpdump -X -i eno2np1.1031 -l -p -nn -e tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eno2np1.1031, link-type EN10MB (Ethernet), snapshot length 262144 bytes 21:22:29.263680 e4:3d:1a:12:b1:00 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 86: 208.80.154.226.46774 > 255.255.255.255.12101: UDP, length 44 0x0000: 4500 0048 903c 4000 4011 3f36 d050 9ae2 E..H.<@.@.?6.P.. 0x0010: ffff ffff b6b6 2f45 0034 0d9a 7465 7374 ....../E.4..test 0x0020: 2062 726f 6164 6361 7374 2069 6e20 766c .broadcast.in.vl 0x0030: 616e 2031 3033 3120 7072 6976 6174 652d an.1031.private- 0x0040: 6531 2d65 7169 6164 e1-eqiad.
Remote Switch (VXLAN tunneled)
Test involves generating a broadcast on an-worker1148, connected to lsw1-f1-eqiad, and confirming receipt on an-worker1147-test, connected to lsw-e1-eqiad. Test frame will be transmitted in Vlan1035 / private-f1-eqiad, which is configured on both switches for the purposes of testing.
Transmit side:
root@an-worker1148-test:~# echo "test broadcast in vlan 1035 private-f1-eqiad" | nc -u -b 255.255.255.255 12345
Receive side:
root@an-worker1147-test:~# tcpdump -X -i eno2np1.1035 -l -p -nn -e tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eno2np1.1035, link-type EN10MB (Ethernet), snapshot length 262144 bytes 21:36:49.189166 e4:3d:1a:54:ab:a7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 87: 10.64.134.11.48282 > 255.255.255.255.12345: UDP, length 45 0x0000: 4500 0049 0a29 4000 4011 a030 0a40 860b E..I.)@.@..0.@.. 0x0010: ffff ffff bc9a 3039 0035 d2a7 7465 7374 ......09.5..test 0x0020: 2062 726f 6164 6361 7374 2069 6e20 766c .broadcast.in.vl 0x0030: 616e 2031 3033 3520 7072 6976 6174 652d an.1035.private- 0x0040: 6631 2d65 7169 6164 0a f1-eqiad.
Client to Client L2 multicast forwarding / ingress replication
Same Switch (Pure L2)
Config the same as 2.7.1 - Client to Client broadcast forwarding / ingress replication
Results
root@ms-fe1012-test:~# echo "test multicast" | nc -q 1 -u -b 224.0.0.26 12345
root@an-worker1147-test:~# tcpdump -e -i eno2np1.1031 -l -nn net 224.0.0.0/8 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eno2np1.1031, link-type EN10MB (Ethernet), snapshot length 262144 bytes 14:43:32.138562 e4:3d:1a:12:b1:00 > 01:00:5e:00:00:1a, ethertype IPv4 (0x0800), length 60: 10.64.130.10.59844 > 224.0.0.26.12345: UDP, length 15
Remote Switch (VXLAN tunneled)
Config the same as 2.7.2 - Client to Client broadcast forwarding / ingress replication
Results
root@an-worker1148-test:~# echo "test multicast evpn type 3" | nc -q 1 -u -b 224.0.0.26 12345
root@an-worker1147-test:~# tcpdump -e -i eno2np1.1035 -l -nn net 224.0.0.0/8 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eno2np1.1035, link-type EN10MB (Ethernet), snapshot length 262144 bytes 17:20:22.025485 e4:3d:1a:54:ab:a7 > 01:00:5e:00:00:1a, ethertype IPv4 (0x0800), length 69: 10.64.134.11.46904 > 224.0.0.26.12345: UDP, length 27
Anycast IP gateway / IRB across multiple switches
When Vlans are stretched across multiple devices 'anycast gws' are configured on each top of rack, with every one acting as gateway for directly connected devices on the Vlan. This test confirms that this is working and the different switches are all responding for this IP.
Config
Tests were done on Vlan1035 / private-f1-eqiad, which is connected to both an-worker1148-test (10.64.134.11) on lsw1-f1-eqiad, and an-worker1147-test (10.64.134.72) on lsw1-e1-eqiad. Layer-2 config the same as in 2.5: Remote MAC Learning on Vlan/VNI.
The IRB interface was configured as follows to provide Anycast GW support:
cmooney@lsw1-e1-eqiad> show configuration interfaces irb unit 1035
virtual-gateway-accept-data;
description private1-f1-eqiad;
family inet {
address 10.64.134.253/24 {
preferred;
virtual-gateway-address 10.64.134.1;
}
}
family inet6 {
address 2620:0:861:10d::253/64 {
preferred;
virtual-gateway-address 2620:0:861:10d::1;
}
}
virtual-gateway-v4-mac 00:00:5e:11:fa:ce;
virtual-gateway-v6-mac 00:00:5e:11:fa:ce;
cmooney@lsw1-f1-eqiad> show configuration interfaces irb unit 1035
virtual-gateway-accept-data;
description private1-f1-eqiad;
family inet {
address 10.64.134.254/24 {
preferred;
virtual-gateway-address 10.64.134.1;
}
}
family inet6 {
address 2620:0:861:10d::254/64 {
preferred;
virtual-gateway-address 2620:0:861:10d::1;
}
}
virtual-gateway-v4-mac 00:00:5e:11:fa:ce;
virtual-gateway-v6-mac 00:00:5e:11:fa:ce;
IPv4
Results
The 3 IPs (one VIP, and 2 per-switch IPs) were pinged from each an-worker1147-test and got a response from each
root@an-worker1147-test:~# ping -c 1 10.64.134.1 PING 10.64.134.1 (10.64.134.1) 56(84) bytes of data. 64 bytes from 10.64.134.1: icmp_seq=1 ttl=64 time=0.677 ms
root@an-worker1147-test:~# ping -c 1 10.64.134.253 PING 10.64.134.253 (10.64.134.253) 56(84) bytes of data. 64 bytes from 10.64.134.253: icmp_seq=1 ttl=64 time=7.93 ms
root@an-worker1147-test:~# ping -c 1 10.64.134.254 PING 10.64.134.254 (10.64.134.254) 56(84) bytes of data. 64 bytes from 10.64.134.254: icmp_seq=1 ttl=64 time=6.06 ms
Doing a monitor on packets processed by lsw1-e1-eqiad we can see that the response from the VIP was processed by it, as well as the ping to the unicast IP:
cmooney@lsw1-e1-eqiad> monitor traffic interface irb.1035 matching "icmp" no-resolve verbose output suppressed, use <detail> or <extensive> for full protocol decode Address resolution is OFF. Listening on irb.1035, capture size 96 bytes 12:25:51.348223 In IP 10.64.134.72 > 10.64.134.1: ICMP echo request, id 46233, seq 1, length 64 12:25:51.348236 Out IP truncated-ip - 52 bytes missing! 10.64.134.1 > 10.64.134.72: ICMP echo reply, id 46233, seq 1, length 64 12:25:57.886469 In IP 10.64.134.72 > 10.64.134.253: ICMP echo request, id 2678, seq 1, length 64 12:25:57.886482 Out IP truncated-ip - 52 bytes missing! 10.64.134.253 > 10.64.134.72: ICMP echo reply, id 2678, seq 1, length 64
The ping to .254, unicast IP on lsw1-f1-eqiad, did not show up in that capture, but instead appears in an equivalent capture on lsw1-f1-eqiad:
cmooney@lsw1-f1-eqiad> monitor traffic interface irb.1035 matching "icmp" no-resolve verbose output suppressed, use <detail> or <extensive> for full protocol decode Address resolution is OFF. Listening on irb.1035, capture size 96 bytes 12:26:00.938116 In IP 10.64.134.72 > 10.64.134.254: ICMP echo request, id 16304, seq 1, length 64
The reverse is the case when the pings are done from an-worker1148-test:
root@an-worker1148-test:~# ping -c 1 10.64.134.1 PING 10.64.134.1 (10.64.134.1) 56(84) bytes of data. 64 bytes from 10.64.134.1: icmp_seq=1 ttl=64 time=14.5 ms
root@an-worker1148-test:~# ping -c 1 10.64.134.253 PING 10.64.134.253 (10.64.134.253) 56(84) bytes of data. 64 bytes from 10.64.134.253: icmp_seq=1 ttl=64 time=14.9 ms
root@an-worker1148-test:~# ping -c 1 10.64.134.254 PING 10.64.134.254 (10.64.134.254) 56(84) bytes of data. 64 bytes from 10.64.134.254: icmp_seq=1 ttl=64 time=6.25 ms
cmooney@lsw1-e1-eqiad> monitor traffic interface irb.1035 matching "icmp" no-resolve verbose output suppressed, use <detail> or <extensive> for full protocol decode Address resolution is OFF. Listening on irb.1035, capture size 96 bytes 12:29:47.721948 In IP 10.64.134.11 > 10.64.134.253: ICMP echo request, id 52386, seq 1, length 64
cmooney@lsw1-f1-eqiad> monitor traffic interface irb.1035 matching "icmp" no-resolve verbose output suppressed, use <detail> or <extensive> for full protocol decode Address resolution is OFF. Listening on irb.1035, capture size 96 bytes 12:29:42.998551 In IP 10.64.134.11 > 10.64.134.1: ICMP echo request, id 375, seq 1, length 64 12:29:42.998564 Out IP truncated-ip - 52 bytes missing! 10.64.134.1 > 10.64.134.11: ICMP echo reply, id 375, seq 1, length 64 12:29:50.194240 In IP 10.64.134.11 > 10.64.134.254: ICMP echo request, id 8113, seq 1, length 64 12:29:50.194252 Out IP truncated-ip - 52 bytes missing! 10.64.134.254 > 10.64.134.11: ICMP echo reply, id 8113, seq 1, length 64
ARP entries appear as expected on both hosts, the VIP shows with the same MAC on both devices:
root@an-worker1147-test:~# ip neigh show dev eno2np1.1035 10.64.134.1 lladdr 00:00:5e:11:fa:ce STALE 10.64.134.11 lladdr e4:3d:1a:54:ab:a7 STALE 10.64.134.253 lladdr a4:e1:1a:81:3a:80 STALE 10.64.134.254 lladdr a4:e1:1a:81:9e:80 STALE
root@an-worker1148-test:~# ip neigh show 10.64.134.1 dev eno2np1 lladdr 00:00:5e:11:fa:ce STALE 10.64.134.10 dev eno2np1 lladdr e4:3d:1a:54:14:45 STALE 10.64.134.72 dev eno2np1 lladdr e4:3d:1a:54:14:45 STALE 10.64.134.253 dev eno2np1 lladdr a4:e1:1a:81:3a:80 STALE 10.64.134.254 dev eno2np1 lladdr a4:e1:1a:81:9e:80 DELAY
IPv6
Results
root@an-worker1147-test:~# ping -c 1 2620:0:861:10d::1 PING 2620:0:861:10d::1(2620:0:861:10d::1) 56 data bytes 64 bytes from 2620:0:861:10d::1: icmp_seq=1 ttl=64 time=0.454 ms
root@an-worker1147-test:~# ping -c 1 2620:0:861:10d::253 PING 2620:0:861:10d::253(2620:0:861:10d::253) 56 data bytes 64 bytes from 2620:0:861:10d::253: icmp_seq=1 ttl=64 time=11.1 ms
root@an-worker1147-test:~# ping -c 1 2620:0:861:10d::254 PING 2620:0:861:10d::254(2620:0:861:10d::254) 56 data bytes 64 bytes from 2620:0:861:10d::254: icmp_seq=1 ttl=64 time=11.2 ms
cmooney@lsw1-e1-eqiad> monitor traffic interface irb.1035 matching "icmp6" no-resolve verbose output suppressed, use <detail> or <extensive> for full protocol decode Address resolution is OFF. Listening on irb.1035, capture size 96 bytes 12:35:29.299922 In IP6 2620:0:861:10d::72 > 2620:0:861:10d::1: ICMP6, echo request, seq 1, length 64 12:35:29.299943 Out [|ip6] 12:35:58.340742 In IP6 2620:0:861:10d::72 > 2620:0:861:10d::253: ICMP6, echo request, seq 1, length 64 12:35:58.340766 Out [|ip6]
cmooney@lsw1-f1-eqiad> monitor traffic interface irb.1035 matching "icmp6" no-resolve verbose output suppressed, use <detail> or <extensive> for full protocol decode Address resolution is OFF. Listening on irb.1035, capture size 96 bytes 12:36:03.678690 In IP6 2620:0:861:10d::72 > 2620:0:861:10d::254: ICMP6, echo request, seq 1, length 64
The same situation is reversed, as expected, doing the same pings from the other host:
root@an-worker1148-test:~# ping -c 1 2620:0:861:10d::1 PING 2620:0:861:10d::1(2620:0:861:10d::1) 56 data bytes 64 bytes from 2620:0:861:10d::1: icmp_seq=1 ttl=64 time=110 ms
root@an-worker1148-test:~# ping -c 1 2620:0:861:10d::253 PING 2620:0:861:10d::253(2620:0:861:10d::253) 56 data bytes 64 bytes from 2620:0:861:10d::253: icmp_seq=1 ttl=64 time=11.3 ms
root@an-worker1148-test:~# ping -c 1 2620:0:861:10d::254 PING 2620:0:861:10d::254(2620:0:861:10d::254) 56 data bytes 64 bytes from 2620:0:861:10d::254: icmp_seq=1 ttl=64 time=11.2 ms
cmooney@lsw1-e1-eqiad> monitor traffic interface irb.1035 matching "icmp6" no-resolve verbose output suppressed, use <detail> or <extensive> for full protocol decode Address resolution is OFF. Listening on irb.1035, capture size 96 bytes 12:40:55.072630 In IP6 2620:0:861:10d::11 > 2620:0:861:10d::253: ICMP6, echo request, seq 1, length 64
cmooney@lsw1-f1-eqiad> monitor traffic interface irb.1035 matching "icmp6" no-resolve verbose output suppressed, use <detail> or <extensive> for full protocol decode Address resolution is OFF. Listening on irb.1035, capture size 96 bytes 12:40:49.343230 In IP6 2620:0:861:10d::11 > 2620:0:861:10d::1: ICMP6, echo request, seq 1, length 64 12:40:49.433829 Out [|ip6] 12:40:58.669189 In IP6 2620:0:861:10d::11 > 2620:0:861:10d::254: ICMP6, echo request, seq 1, length 64 12:40:58.669212 Out [|ip6]
IPv6 Neighbors are as expected on both hosts:
root@an-worker1147-test:~# ip -6 neigh show dev eno2np1.1035 2620:0:861:10d::1 lladdr 00:00:5e:11:fa:ce router STALE 2620:0:861:10d::253 lladdr a4:e1:1a:81:3a:80 router STALE 2620:0:861:10d::254 lladdr a4:e1:1a:81:9e:80 router STALE
root@an-worker1148-test:~# ip -6 neigh show 2620:0:861:10d::1 dev eno2np1 lladdr 00:00:5e:11:fa:ce router STALE 2620:0:861:10d::253 dev eno2np1 lladdr a4:e1:1a:81:3a:80 router STALE 2620:0:861:10d::254 dev eno2np1 lladdr a4:e1:1a:81:9e:80 router STALE
Inter-Vlan/subnet routing via IRB interfaces on same switch
Tests carried out on lsw1-e1-eqiad, between the IRB interfaces assocatied with private1-e1-eqiad and analytics-e1-eqiad (no filter in place for test).
Relevant Configuration
Relevant Config'
private1-e1-eqiad / ms-fe1012-test:
cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1031
description private1-e1-eqiad;
family inet {
address 10.64.130.1/24;
}
family inet6 {
address 2620:0:861:109::1/64;
}
root@ms-fe1012-test:~# ip -br addr show dev enp101s0f0np0 enp101s0f0np0 UP 10.64.130.10/24 2620:0:861:109::10/64 fe80::e63d:1aff:fe12:b100/64
root@ms-fe1012-test:~# ip route show | grep default default via 10.64.130.1 dev enp101s0f0np0
root@ms-fe1012-test:~# ip -6 route show | grep default default via 2620:0:861:109::1 dev enp101s0f0np0 metric 1024 pref medium
analytics1-e1-eqiad / an-worker1148-test
cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1039
description analytics1-e1-eqiad;
family inet {
address 10.64.138.1/24;
}
family inet6 {
address 2620:0:861:100::1/64;
}
root@an-worker1147-test:/etc# ip -br addr show dev eno2np1 eno2np1 UP 10.64.138.11/24 2620:0:861:100::11/64 fe80::e63d:1aff:fe54:1445/64
root@an-worker1147-test:/etc# ip route show | grep default default via 10.64.138.1 dev eno2np1
root@an-worker1147-test:/etc# ip -6 route show | grep default default via 2620:0:861:100::1 dev eno2np1 metric 1024 pref medium
IPv4
Results
root@ms-fe1012-test:~# mtr --address 10.64.130.10 -b -w -c 5 10.64.138.11
Start: 2022-02-10T17:07:22+0000
HOST: ms-fe1012-test Loss% Snt Last Avg Best Wrst StDev
1.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (10.64.130.1) 0.0% 5 6.2 6.0 2.3 8.2 2.2
2.|-- 10.64.138.11 0.0% 5 0.3 0.2 0.2 0.3 0.0
root@an-worker1147-test:~# mtr --address 10.64.138.11 -b -w -c 5 10.64.130.10
Start: 2022-02-10T17:06:33+0000
HOST: an-worker1147-test Loss% Snt Last Avg Best Wrst StDev
1.|-- irb-1039.lsw1-e1-eqiad.eqiad.wmnet (10.64.138.1) 0.0% 5 5.7 4.5 0.8 9.9 3.6
2.|-- 10.64.130.10 0.0% 5 0.2 0.2 0.2 0.2 0.0
IPv6
Results
root@ms-fe1012-test:~# mtr --address 2620:0:861:109::10 -b -w -c 5 2620:0:861:100::11
Start: 2022-02-10T17:08:11+0000
HOST: ms-fe1012-test Loss% Snt Last Avg Best Wrst StDev
1.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:109::1) 0.0% 5 0.5 0.5 0.5 0.6 0.0
2.|-- 2620:0:861:100::11 0.0% 5 0.2 0.2 0.2 0.2 0.0
root@an-worker1147-test:~# mtr --address 2620:0:861:100::11 -b -w -c 5 2620:0:861:109::10
Start: 2022-02-10T17:08:21+0000
HOST: an-worker1147-test Loss% Snt Last Avg Best Wrst StDev
1.|-- irb-1039.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:100::1) 0.0% 5 0.6 0.5 0.5 0.6 0.0
2.|-- 2620:0:861:109::10 0.0% 5 0.2 0.2 0.2 0.2 0.0
Inter-Vlan/subnet routing via IRB interfaces on separate switches
Tests carried out between ms-fe1012-test, connected to private1-e1-eqiad on lsw1-e1-eqiad, and an-worker1148-test, connected to analytics1-f1-eqiad on lsw1-f1-eqiad.
Relevant Config
private1-e1-eqiad / ms-fe1012-test:
cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1031
description private1-e1-eqiad;
family inet {
address 10.64.130.1/24;
}
family inet6 {
address 2620:0:861:109::1/64;
}
root@ms-fe1012-test:~# ip -br addr show dev enp101s0f0np0 enp101s0f0np0 UP 10.64.130.10/24 2620:0:861:109::10/64 fe80::e63d:1aff:fe12:b100/64
root@ms-fe1012-test:~# ip route show | grep default default via 10.64.130.1 dev enp101s0f0np0
root@ms-fe1012-test:~# ip -6 route show | grep default default via 2620:0:861:109::1 dev enp101s0f0np0 metric 1024 pref medium
analytics1-f1-eqiad / an-worker1148-test:
cmooney@lsw1-f1-eqiad> show configuration interfaces irb.1043
description analytics1-f1-eqiad;
family inet {
address 10.64.142.1/24;
}
family inet6 {
address 2620:0:861:114::1/64;
}
root@an-worker1148-test:~# ip -br addr show dev eno2np1.1043 eno2np1.1043@eno2np1 UP 10.64.142.10/24 2620:0:861:114::10/64 fe80::e63d:1aff:fe54:aba7/64
root@an-worker1148-test:~# ip route show | grep 10.0.0.0 10.0.0.0/8 via 10.64.142.1 dev eno2np1.1043
root@an-worker1148-test:~# ip -6 route show | grep default default via 2620:0:861:114::1 dev eno2np1.1043 metric 1024 pref medium
IPv4
Results
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 10.64.142.0 detail
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
10.64.142.0/24 (1 entry, 1 announced)
*EVPN Preference: 170/-101
Next hop type: Indirect, Next hop index: 0
Address: 0xc686b6c
Next-hop reference count: 15
Next hop type: Router, Next hop index: 1765
Next hop: 10.64.129.7 via et-0/0/52.0, selected
Session Id: 0x0
Protocol next hop: 10.64.128.7
Composite next hop: 0xbd31530 1760 INH Session ID: 0x0
VXLAN tunnel rewrite:
MTU: 0, Flags: 0x0
Encap table ID: 0, Decap table ID: 10
Encap VNI: 3005000, Decap VNI: 3005000
Source VTEP: 10.64.128.3, Destination VTEP: 10.64.128.7
SMAC: a4:e1:1a:81:3a:80, DMAC: a4:e1:1a:81:9e:80
Indirect next hop: 0xc816e84 524299 INH Session ID: 0x0
State: <Active Int Ext>
Age: 2d 0:43:19 Metric2: 8
Validation State: unverified
Task: PRODUCTION-EVPN-L3-context
Announcement bits (2): 2-KRT 4-BGP_RT_Background
AS path: I
Thread: junos-main
cmooney@lsw1-f1-eqiad> show route table PRODUCTION.inet.0 10.64.130.0 detail
PRODUCTION.inet.0: 34 destinations, 37 routes (34 active, 0 holddown, 0 hidden)
10.64.130.0/24 (1 entry, 1 announced)
*EVPN Preference: 170/-101
Next hop type: Indirect, Next hop index: 0
Address: 0xc68733c
Next-hop reference count: 21
Next hop type: Router, Next hop index: 1748
Next hop: 10.64.129.6 via et-0/0/52.0, selected
Session Id: 0x0
Protocol next hop: 10.64.128.3
Composite next hop: 0xbd309a0 1751 INH Session ID: 0x0
VXLAN tunnel rewrite:
MTU: 0, Flags: 0x0
Encap table ID: 0, Decap table ID: 8
Encap VNI: 3005000, Decap VNI: 3005000
Source VTEP: 10.64.128.7, Destination VTEP: 10.64.128.3
SMAC: a4:e1:1a:81:9e:80, DMAC: a4:e1:1a:81:3a:80
Indirect next hop: 0xc815b04 524294 INH Session ID: 0x0
State: <Active Int Ext>
Age: 6d 22:20:29 Metric2: 8
Validation State: unverified
Task: PRODUCTION-EVPN-L3-context
Announcement bits (2): 2-KRT 3-BGP_RT_Background
AS path: I
Thread: junos-main
root@ms-fe1012-test:~# mtr --address 10.64.130.10 -c 5 -b -w 10.64.142.10
Start: 2022-02-10T17:18:41+0000
HOST: ms-fe1012-test Loss% Snt Last Avg Best Wrst StDev
1.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (10.64.130.1) 0.0% 5 3.1 4.7 1.4 7.6 2.6
2.|-- irb-1043.lsw1-f1-eqiad.eqiad.wmnet (10.64.142.1) 0.0% 5 2.2 3.6 0.9 6.9 2.3
3.|-- 10.64.142.10 0.0% 5 0.2 0.2 0.2 0.2 0.0
root@an-worker1148-test:~# mtr --address 10.64.142.10 -c 5 -b -w 10.64.130.10
Start: 2022-02-10T16:58:30+0000
HOST: an-worker1148-test Loss% Snt Last Avg Best Wrst StDev
1.|-- irb-1043.lsw1-f1-eqiad.eqiad.wmnet (10.64.142.1) 0.0% 5 3.7 6.8 3.7 8.4 1.9
2.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (10.64.130.1) 0.0% 5 7.2 4.5 1.2 7.2 2.5
3.|-- 10.64.130.10 0.0% 5 0.2 0.2 0.2 0.3 0.0
IPv6
Results
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 2620:0:861:114:: detail
PRODUCTION.inet6.0: 40 destinations, 44 routes (40 active, 0 holddown, 0 hidden)
2620:0:861:114::/64 (1 entry, 1 announced)
*EVPN Preference: 170/-101
Next hop type: Indirect, Next hop index: 0
Address: 0xc6862d4
Next-hop reference count: 13
Next hop type: Router, Next hop index: 1765
Next hop: 10.64.129.7 via et-0/0/52.0, selected
Session Id: 0x0
Protocol next hop: 10.64.128.7
Composite next hop: 0xbd314e0 1759 INH Session ID: 0x0
VXLAN tunnel rewrite:
MTU: 0, Flags: 0x0
Encap table ID: 0, Decap table ID: 10
Encap VNI: 3005000, Decap VNI: 3005000
Source VTEP: 10.64.128.3, Destination VTEP: 10.64.128.7
SMAC: a4:e1:1a:81:3a:80, DMAC: a4:e1:1a:81:9e:80
Indirect next hop: 0xc816d04 524298 INH Session ID: 0x0
State: <Active Int Ext>
Age: 2d 0:46:31 Metric2: 8
Validation State: unverified
Task: PRODUCTION-EVPN-L3-context
Announcement bits (2): 2-KRT 4-BGP_RT_Background
AS path: I
Thread: junos-main
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 2620:0:861:114:: detail
PRODUCTION.inet6.0: 37 destinations, 41 routes (37 active, 0 holddown, 0 hidden)
2620:0:861:109::/64 (1 entry, 1 announced)
*EVPN Preference: 170/-101
Next hop type: Indirect, Next hop index: 0
Address: 0xc6865f4
Next-hop reference count: 19
Next hop type: Router, Next hop index: 1748
Next hop: 10.64.129.6 via et-0/0/52.0, selected
Session Id: 0x0
Protocol next hop: 10.64.128.3
Composite next hop: 0xbd30950 1750 INH Session ID: 0x0
VXLAN tunnel rewrite:
MTU: 0, Flags: 0x0
Encap table ID: 0, Decap table ID: 8
Encap VNI: 3005000, Decap VNI: 3005000
Source VTEP: 10.64.128.7, Destination VTEP: 10.64.128.3
SMAC: a4:e1:1a:81:9e:80, DMAC: a4:e1:1a:81:3a:80
Indirect next hop: 0xc815984 524293 INH Session ID: 0x0
State: <Active Int Ext>
Age: 6d 22:23:45 Metric2: 8
Validation State: unverified
Task: PRODUCTION-EVPN-L3-context
Announcement bits (2): 2-KRT 3-BGP_RT_Background
AS path: I
Thread: junos-main
root@ms-fe1012-test:~# mtr --address 2620:0:861:109::10 -b -w -c 5 2620:0:861:114::10
Start: 2022-02-10T17:18:04+0000
HOST: ms-fe1012-test Loss% Snt Last Avg Best Wrst StDev
1.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:109::1) 0.0% 5 0.5 0.5 0.5 0.5 0.0
2.|-- irb-1043.lsw1-f1-eqiad.eqiad.wmnet (2620:0:861:114::1) 0.0% 5 0.5 0.5 0.4 0.6 0.1
3.|-- 2620:0:861:114::10 0.0% 5 0.3 0.3 0.3 0.3 0.0
root@an-worker1148-test:~# mtr --address 2620:0:861:114::10 -c 5 -b -w 2620:0:861:109::10
Start: 2022-02-10T17:00:30+0000
HOST: an-worker1148-test Loss% Snt Last Avg Best Wrst StDev
1.|-- irb-1043.lsw1-f1-eqiad.eqiad.wmnet (2620:0:861:114::1) 0.0% 5 0.7 9.0 0.5 42.6 18.8
2.|-- irb-1031.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:109::1) 0.0% 5 0.6 0.5 0.5 0.6 0.0
3.|-- 2620:0:861:109::10 0.0% 5 0.2 0.2 0.2 0.3 0.0
BGP Peering on Vlan segment to end device
These tests verify that end hosts can successfully create a BGP peering to the switch IRB interface of their connected subnet. This first set of tests validates the normal scenario where there is a per-rack subnet, and thus an IRB interface with only a single unicast IP.
Tests were done with ms-fe1012-test, connected to lsw1-e1-eqiad on Vlan1031 / private1-e1-eqiad. A public IP addresses (208.80.154.226/32 and 2620:0:861:ed1a::3/128) were allocated from the LVS public range and the peering defined as a standard LVS one in automation.
Relevant Config
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group PyBal
type external;
hold-time 30;
import LVS_IMPORT;
family inet {
unicast {
prefix-limit {
maximum 1000;
teardown 20;
}
}
}
family inet6 {
unicast {
prefix-limit {
maximum 1000;
teardown 20;
}
}
}
export NONE;
peer-as 64600;
neighbor 10.64.130.10 {
description ms-fe1012-test;
}
IPv4 only
cmooney@lsw1-e1-eqiad> show bgp neighbor 10.64.130.10
Peer: 10.64.130.10+41482 AS 64600 Local: 10.64.130.1+179 AS 64810
Description: ms-fe1012-test
Group: PyBal Routing-Instance: PRODUCTION
Forwarding routing-instance: PRODUCTION
Type: External State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
Export: [ NONE ] Import: [ LVS_IMPORT ]
Options: <Preference HoldTime AddressFamily PeerAS PrefixLimit LocalAS Refresh>
Options: <GracefulShutdownRcv>
Address families configured: inet-unicast inet6-unicast
Holdtime: 30 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Local AS: 64810 Local System AS: 64810
Number of flaps: 5
Last flap event: RecvNotify
Error: 'Cease' Sent: 0 Recv: 5
Peer ID: 208.80.154.226 Local ID: 10.64.130.1 Active Holdtime: 30
Keepalive Interval: 10 Group index: 10 Peer index: 0 SNMP index: 28
I/O Session Thread: bgpio-0 State: Enabled
BFD: disabled, down
Local Interface: irb.1031
NLRI for restart configured on peer: inet-unicast inet6-unicast
NLRI advertised by peer: inet-unicast
NLRI for this session: inet-unicast
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
NLRI that restart is negotiated for: inet-unicast
NLRI of received end-of-rib markers: inet-unicast
NLRI of all end-of-rib markers sent: inet-unicast
Peer does not support LLGR Restarter or Receiver functionality
Peer supports 4 byte AS extension (peer-as 64600)
NLRI's for which peer can receive multiple paths: inet-unicast
Table PRODUCTION.inet.0 Bit: 90001
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: in sync
Active prefixes: 1
Received prefixes: 1
Accepted prefixes: 1
Suppressed due to damping: 0
Advertised prefixes: 0
Last traffic (seconds): Received 2 Sent 2 Checked 252
Input messages: Total 29 Updates 2 Refreshes 0 Octets 668
Output messages: Total 28 Updates 0 Refreshes 0 Octets 536
Output Queue[8]: 0 (PRODUCTION.inet.0, inet-unicast)
cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.130.10 table PRODUCTION.inet.0
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 208.80.154.226/32 10.64.130.10 0 64600Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 10.64.130.1 count 100 rapid 208.80.154.226 PING 208.80.154.226 (208.80.154.226): 56 data bytes  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! --- 208.80.154.226 ping statistics --- 100 packets transmitted, 100 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.298/7.792/26.546/2.866 ms
IPv4 carrying IPv4 & IPv6 address families
cmooney@lsw1-e1-eqiad> show bgp neighbor 10.64.130.10
Peer: 10.64.130.10+41484 AS 64600 Local: 10.64.130.1+179 AS 64810
Description: ms-fe1012-test
Group: PyBal Routing-Instance: PRODUCTION
Forwarding routing-instance: PRODUCTION
Type: External State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
Export: [ NONE ] Import: [ LVS_IMPORT ]
Options: <Preference HoldTime AddressFamily PeerAS PrefixLimit LocalAS Refresh>
Options: <GracefulShutdownRcv>
Address families configured: inet-unicast inet6-unicast
Holdtime: 30 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Local AS: 64810 Local System AS: 64810
Number of flaps: 6
Last flap event: RecvNotify
Error: 'Cease' Sent: 0 Recv: 6
Peer ID: 208.80.154.226 Local ID: 10.64.130.1 Active Holdtime: 30
Keepalive Interval: 10 Group index: 10 Peer index: 0 SNMP index: 28
I/O Session Thread: bgpio-0 State: Enabled
BFD: disabled, down
Local Interface: irb.1031
NLRI for restart configured on peer: inet-unicast inet6-unicast
NLRI advertised by peer: inet-unicast inet6-unicast
NLRI for this session: inet-unicast inet6-unicast
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
NLRI that restart is negotiated for: inet-unicast inet6-unicast
NLRI of received end-of-rib markers: inet-unicast inet6-unicast
NLRI of all end-of-rib markers sent: inet-unicast inet6-unicast
Peer does not support LLGR Restarter or Receiver functionality
Peer supports 4 byte AS extension (peer-as 64600)
NLRI's for which peer can receive multiple paths: inet-unicast inet6-unicast
Table PRODUCTION.inet.0 Bit: 90001
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: in sync
Active prefixes: 1
Received prefixes: 1
Accepted prefixes: 1
Suppressed due to damping: 0
Advertised prefixes: 0
Table PRODUCTION.inet6.0 Bit: a0001
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: in sync
Active prefixes: 0
Received prefixes: 0
Accepted prefixes: 0
Suppressed due to damping: 0
Advertised prefixes: 0
Last traffic (seconds): Received 3 Sent 4 Checked 13
Input messages: Total 6 Updates 3 Refreshes 0 Octets 253
Output messages: Total 4 Updates 0 Refreshes 0 Octets 91
Output Queue[8]: 0 (PRODUCTION.inet.0, inet-unicast)
Output Queue[9]: 0 (PRODUCTION.inet6.0, inet6-unicast)
cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.130.10 table PRODUCTION.inet.0
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 208.80.154.226/32 10.64.130.10 0 64600Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 10.64.130.1 count 100 rapid 208.80.154.226 PING 208.80.154.226 (208.80.154.226): 56 data bytes  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.130.10 table PRODUCTION.inet6.0
PRODUCTION.inet6.0: 40 destinations, 44 routes (40 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
2620:0:861:ed1a::3/128
* 2620:0:861:109::10 0 64600Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 2620:0:861:109::1 count 100 rapid 2620:0:861:ed1a::3 PING6(56=40+8+8 bytes) 2620:0:861:109::1 --> 2620:0:861:ed1a::3 Â !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
IPv4 & IPv6 each carrying their own address family
These tests were instead carried out from an-worker1147-test to lsw1-e1-eqiad. Two BGP peerings are defined towards the host, one in the 'Kubernetes4' group and the other in 'Kubernetes6'. Peering was within Vlan1039 / analytics1-e1-eqiad for both address types.
Relevant Config
cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1039
description analytics1-e1-eqiad;
family inet {
address 10.64.138.1/24;
}
family inet6 {
address 2620:0:861:100::1/64;
}
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group Kubernetes4
type external;
multihop {
ttl 2;
}
hold-time 30;
import kubernetes_import;
family inet {
unicast {
prefix-limit {
maximum 2000;
teardown 80;
}
}
}
export NONE;
peer-as 64601;
multipath;
neighbor 10.64.138.11 {
description an-worker1147-test;
}
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group Kubernetes6
type external;
multihop {
ttl 2;
}
hold-time 30;
import kubernetes_import;
family inet6 {
unicast {
prefix-limit {
maximum 2000;
teardown 80;
}
}
}
export NONE;
peer-as 64601;
multipath;
neighbor 2620:0:861:100::11 {
description an-worker1147-test;
}
Results
cmooney@lsw1-e1-eqiad> show bgp neighbor 10.64.138.11
Peer: 10.64.138.11+60646 AS 64601 Local: 10.64.138.1+179 AS 64810
Description: an-worker1147-test
Group: Kubernetes4 Routing-Instance: PRODUCTION
Forwarding routing-instance: PRODUCTION
Type: External State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: Refresh
Last Error: None
Export: [ NONE ] Import: [ kubernetes_import ]
Options: <Multihop Preference HoldTime Ttl AddressFamily PeerAS Multipath PrefixLimit LocalAS Refresh>
Options: <GracefulShutdownRcv>
Address families configured: inet-unicast
Holdtime: 30 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Local AS: 64810 Local System AS: 64810
Number of flaps: 0
Peer ID: 208.80.154.228 Local ID: 10.64.130.1 Active Holdtime: 30
Keepalive Interval: 10 Group index: 12 Peer index: 0 SNMP index: 29
I/O Session Thread: bgpio-0 State: Enabled
BFD: disabled, down
NLRI for restart configured on peer: inet-unicast
NLRI advertised by peer: inet-unicast
NLRI for this session: inet-unicast
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
NLRI that restart is negotiated for: inet-unicast
NLRI of received end-of-rib markers: inet-unicast
NLRI of all end-of-rib markers sent: inet-unicast
Peer does not support LLGR Restarter or Receiver functionality
Peer supports 4 byte AS extension (peer-as 64601)
NLRI's for which peer can receive multiple paths: inet-unicast
Table PRODUCTION.inet.0 Bit: 90002
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: in sync
Active prefixes: 0
Received prefixes: 1
Accepted prefixes: 0
Suppressed due to damping: 0
Advertised prefixes: 0
Last traffic (seconds): Received 5 Sent 6 Checked 615
Input messages: Total 69 Updates 5 Refreshes 1 Octets 1520
Output messages: Total 69 Updates 0 Refreshes 0 Octets 1315
Output Queue[8]: 0 (PRODUCTION.inet.0, inet-unicast)
cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.138.11 table PRODUCTION.inet.0
PRODUCTION.inet.0: 35 destinations, 39 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
208.80.154.228/32 10.64.138.11 0 64601Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 10.64.138.1 count 100 rapid 208.80.154.228 PING 208.80.154.228 (208.80.154.228): 56 data bytes  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cmooney@lsw1-e1-eqiad> show bgp neighbor 2620:0:861:100::11
Peer: 2620:0:861:100::11+41120 AS 64601 Local: 2620:0:861:100::1+179 AS 64810
Description: an-worker1147-test
Group: Kubernetes6 Routing-Instance: PRODUCTION
Forwarding routing-instance: PRODUCTION
Type: External State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: Refresh
Last Error: Open Message Error
Export: [ NONE ] Import: [ kubernetes_import ]
Options: <Multihop Preference HoldTime Ttl AddressFamily PeerAS Multipath PrefixLimit LocalAS Refresh>
Options: <GracefulShutdownRcv>
Address families configured: inet6-unicast
Holdtime: 30 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Local AS: 64810 Local System AS: 64810
Number of flaps: 0
Error: 'Open Message Error' Sent: 4 Recv: 0
Peer ID: 208.80.154.228 Local ID: 10.64.130.1 Active Holdtime: 30
Keepalive Interval: 10 Group index: 13 Peer index: 0 SNMP index: 30
I/O Session Thread: bgpio-0 State: Enabled
BFD: disabled, down
NLRI for restart configured on peer: inet6-unicast
NLRI advertised by peer: inet6-unicast
NLRI for this session: inet6-unicast
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
NLRI that restart is negotiated for: inet6-unicast
NLRI of received end-of-rib markers: inet6-unicast
NLRI of all end-of-rib markers sent: inet6-unicast
Peer does not support LLGR Restarter or Receiver functionality
Peer supports 4 byte AS extension (peer-as 64601)
NLRI's for which peer can receive multiple paths: inet6-unicast
Table PRODUCTION.inet6.0 Bit: a0002
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: in sync
Active prefixes: 0
Received prefixes: 1
Accepted prefixes: 0
Suppressed due to damping: 0
Advertised prefixes: 0
Last traffic (seconds): Received 8 Sent 1 Checked 348
Input messages: Total 43 Updates 6 Refreshes 1 Octets 1256
Output messages: Total 39 Updates 0 Refreshes 0 Octets 752
Output Queue[9]: 0 (PRODUCTION.inet6.0, inet6-unicast)
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:100::11
PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
2620:0:861:ed1a::4/128
* 2620:0:861:100::11 0 64601Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 2620:0:861:100::1 count 100 rapid 2620:0:861:ed1a::4 PING6(56=40+8+8 bytes) 2620:0:861:100::1 --> 2620:0:861:ed1a::4 Â !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
BGP Peering on Vlan segment from Anycast GW IP
BGP peering for these tests was carried out from an-worker1147-test, connected to lsw1-e1-eqiad. Vlan private1-f1-eqiad was extended to this switch, and an Anycast GW configuration added to the IRB interface to support this.
Relevant Config
cmooney@lsw1-e1-eqiad> show configuration interfaces irb.1035
virtual-gateway-accept-data;
description private1-f1-eqiad;
family inet {
address 10.64.134.253/24 {
preferred;
virtual-gateway-address 10.64.134.1;
}
}
family inet6 {
address 2620:0:861:10d::253/64 {
preferred;
virtual-gateway-address 2620:0:861:10d::1;
}
}
virtual-gateway-v4-mac 00:00:5e:11:fa:ce;
virtual-gateway-v6-mac 00:00:5e:11:fa:ce;
IPv4 only
Peering was done by adding the IP address of an-worker1147-test on Vlan1135 to the "Anycast4" BGP group:
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group Anycast4
type external;
multihop {
ttl 193;
}
damping;
import ANYCAST_IMPORT;
family inet {
unicast {
prefix-limit {
maximum 50;
teardown 80;
}
}
}
export NONE;
peer-as 64605;
multipath;
bfd-liveness-detection {
minimum-interval 300;
}
neighbor 10.64.134.72 {
description an-worker1147-test;
}
Connection was established from the an-worker1147-test IP on Vlan1135, to Anycast IP 10.64.134.1 (using FRR BGP stack), public IP was announced:
an-worker1147-test# show bgp ipv4 unicast neighbors 10.64.134.1 advertised-routes
BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
Default local pref 100, local AS 64605
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 208.80.154.228/32
0.0.0.0 0 32768Â ?
Route is accepted by the top of rack and public IP is pingable:
cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet.0
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 208.80.154.228/32 10.64.134.72 0 64605Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION rapid count 100 source 10.64.134.1 208.80.154.228 PING 208.80.154.228 (208.80.154.228): 56 data bytes  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
Start: 2022-02-14T17:18:04+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6830 176.61.34.1 0.0% 10 14.5 15.0 11.9 20.5 2.8
2. AS6830 109.255.255.254 0.0% 10 11.3 16.4 7.6 55.0 13.8
3. AS6830 ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42) 0.0% 10 10.1 15.9 10.1 30.4 6.0
4. AS6830 ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110) 0.0% 10 10.6 15.4 9.0 45.5 10.8
5. AS1299 dln-b2-link.ip.twelve99.net (62.115.172.136) 0.0% 10 19.1 13.3 7.7 20.4 4.2
6. AS1299 ldn-bb1-link.ip.twelve99.net (62.115.120.100) 0.0% 10 20.3 21.5 15.7 26.3 3.0
7. AS1299 nyk-bb2-link.ip.twelve99.net (62.115.113.20) 0.0% 10 94.7 97.3 94.4 100.2 2.4
8. AS1299 ash-bb2-link.ip.twelve99.net (62.115.136.201) 0.0% 10 99.2 101.2 97.1 105.5 3.1
9. AS1299 ash-b1-link.ip.twelve99.net (62.115.143.121) 0.0% 10 101.7 103.3 100.6 108.5 2.5
10. AS1299 wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226) 0.0% 10 101.4 99.7 95.6 103.2 2.4
11. AS14907 ae0.cr1-eqiad.wikimedia.org (208.80.154.193) 0.0% 10 105.5 103.1 97.0 110.5 5.0
12. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
13. AS14907 208.80.154.228 0.0% 10 101.4 99.4 94.8 104.6 3.0
IPv4 carrying IPv4 & IPv6 address families
For this test the IP of an-worker1147-test on Vlan1035 was placed into the 'PyBal' group in the overlay BGP config, which means the switch will announce the capability for IPv4 and IPv6 address families over the single IPv4 peering.
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group PyBal
type external;
hold-time 30;
import LVS_IMPORT;
family inet {
unicast {
prefix-limit {
maximum 1000;
teardown 20;
}
}
}
family inet6 {
unicast {
prefix-limit {
maximum 1000;
teardown 20;
}
}
}
export NONE;
peer-as 64600;
neighbor 10.64.134.72 {
description an-worker1147-test;
}
Host was configured to send both types of route over the session:
an-worker1147-test# show bgp ipv4 neighbors 10.64.134.1 advertised-routes
BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
Default local pref 100, local AS 64600
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 208.80.154.228/32
0.0.0.0 0 32768Â ?
an-worker1147-test# show bgp ipv6 neighbors 10.64.134.1 advertised-routes
BGP table version is 3, local router ID is 208.80.154.228, vrf id 0
Default local pref 100, local AS 64600
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 2620:0:861:ed1a::4/128
 :: 0 32768 ?
IPv4 received and works:
cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet.0
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 208.80.154.228/32 10.64.134.72 0 64600Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 208.80.154.228 rapid count 100 source 10.64.134.253 PING 208.80.154.228 (208.80.154.228): 56 data bytes  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
Start: 2022-02-14T17:42:02+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6830 176.61.34.1 0.0% 10 13.5 16.0 10.3 26.0 4.8
2. AS6830 109.255.255.254 0.0% 10 17.2 13.3 9.8 17.2 2.4
3. AS6830 ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42) 0.0% 10 11.6 12.3 8.9 17.1 2.9
4. AS6830 ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110) 0.0% 10 28.3 13.4 9.0 28.3 5.7
5. AS1299 dln-b2-link.ip.twelve99.net (62.115.172.136) 0.0% 10 13.2 11.8 9.1 14.3 1.8
6. AS1299 ldn-bb1-link.ip.twelve99.net (62.115.120.100) 0.0% 10 22.4 23.3 19.0 33.0 3.7
7. AS1299 nyk-bb2-link.ip.twelve99.net (62.115.113.20) 0.0% 10 96.4 97.9 92.7 105.9 4.0
8. AS1299 ash-bb2-link.ip.twelve99.net (62.115.136.201) 0.0% 10 100.2 102.6 93.4 125.7 9.2
9. AS1299 ash-b1-link.ip.twelve99.net (62.115.143.121) 0.0% 10 98.5 102.6 97.1 106.7 3.1
10. AS1299 wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226) 0.0% 10 100.0 101.5 98.3 105.5 2.8
11. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
12. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
13. AS14907 208.80.154.228 0.0% 10 98.0 98.5 94.9 106.3 3.5
IPv6 also received and works:
cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet6.0
PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
2620:0:861:ed1a::4/128
* 2620:0:861:10d::72 0 64600Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 2620:0:861:ed1a::4 rapid count 100 source 2620:0:861:10d::253 PING6(56=40+8+8 bytes) 2620:0:861:10d::253 --> 2620:0:861:ed1a::4 Â !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cathal@nbgw:~$ mtr -b -w -z -c 10 2620:0:861:ed1a::4
Start: 2022-02-14T17:39:50+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6939 tunnel650354.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:32c::1) 10.0% 10 23.9 28.5 22.1 43.7 7.8
2. AS6939 e0-19.core2.lon2.he.net (2001:470:0:67::1) 0.0% 10 35.8 41.3 23.8 110.6 28.7
3. AS6939 100ge11-2.core1.lon2.he.net (2001:470:0:541::1) 0.0% 10 24.7 29.0 20.1 61.5 14.2
4. AS6939 100ge4-1.core1.nyc4.he.net (2001:470:0:2cf::2) 0.0% 10 103.7 88.7 81.4 103.7 6.1
5. AS6939 100ge11-1.core1.nyc5.he.net (2001:470:0:20a::2) 0.0% 10 90.4 93.2 84.8 138.5 16.2
6. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
7. AS6939 100ge1-2.core1.ash1.he.net (2001:470:0:277::1) 0.0% 10 108.1 114.2 94.7 188.9 26.9
8. AS6939 xe-5-3-3-500.cr1-eqiad.wikimedia.org (2001:470:0:1c0::2) 0.0% 10 96.3 101.0 90.3 130.5 14.0
9. AS14907 et-0-0-48-100.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:fe07::2) 10.0% 10 94.6 101.7 92.6 155.8 20.3
10. AS14907 2620:0:861:ed1a::4 0.0% 10 110.5 103.3 89.6 140.9 16.9
IPv4 & IPv6 each carrying their own address family
For the final test separate BGP peering sessions to an-worker1147-test were established from lsw1-e1-eqiad, over both IPv4 and IPv6, exchanging just the routes for a single address family over each.
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group Kubernetes4
type external;
multihop {
ttl 2;
}
hold-time 30;
import kubernetes_import;
family inet {
unicast {
prefix-limit {
maximum 2000;
teardown 80;
}
}
}
export NONE;
peer-as 64601;
multipath;
neighbor 10.64.134.72 {
description an-worker1147-test;
}
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group Kubernetes6
type external;
multihop {
ttl 2;
}
hold-time 30;
import kubernetes_import;
family inet6 {
unicast {
prefix-limit {
maximum 2000;
teardown 80;
}
}
}
export NONE;
peer-as 64601;
multipath;
neighbor 2620:0:861:10d::72 {
description an-worker1147-test;
}
IPv4 results:
an-worker1147-test# show bgp ipv4 neighbors 10.64.134.1 advertised-routes
BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
Default local pref 100, local AS 64601
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 208.80.154.228/32
0.0.0.0 0 32768Â ?
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.64.134.72
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 208.80.154.228/32 10.64.134.72 0 64601Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 208.80.154.228 rapid count 100 source 10.64.134.253 PING 208.80.154.228 (208.80.154.228): 56 data bytes  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
Start: 2022-02-14T18:10:25+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6830 176.61.34.1 0.0% 10 18.3 15.4 10.1 20.6 3.8
2. AS6830 109.255.255.254 0.0% 10 16.0 18.2 8.5 50.2 12.2
3. AS6830 ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42) 0.0% 10 15.2 19.1 10.4 51.6 12.3
4. AS6830 ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110) 0.0% 10 12.9 17.8 8.5 53.9 13.0
5. AS1299 dln-b2-link.ip.twelve99.net (62.115.172.136) 0.0% 10 15.9 14.6 9.5 19.5 3.0
6. AS1299 ldn-bb1-link.ip.twelve99.net (62.115.120.100) 0.0% 10 23.3 28.3 20.1 53.7 10.0
7. AS1299 nyk-bb2-link.ip.twelve99.net (62.115.113.20) 0.0% 10 106.9 98.9 94.2 106.9 4.2
8. AS1299 ash-bb2-link.ip.twelve99.net (62.115.136.201) 0.0% 10 112.7 105.3 98.4 120.2 6.9
9. AS1299 ash-b1-link.ip.twelve99.net (62.115.143.121) 0.0% 10 104.5 106.9 101.0 125.8 7.6
10. AS1299 wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226) 0.0% 10 111.7 106.2 96.1 112.7 6.0
11. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
12. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
13. AS14907 208.80.154.228 0.0% 10 104.1 106.2 93.8 125.3 11.0
IPv6 routes:
an-worker1147-test# show bgp ipv6 neighbors 2620:0:861:10d::1 advertised-routes
BGP table version is 3, local router ID is 208.80.154.228, vrf id 0
Default local pref 100, local AS 64601
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 2620:0:861:ed1a::4/128
 :: 0 32768 ?
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:10d::72
PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
2620:0:861:ed1a::4/128
* 2620:0:861:10d::72 0 64601Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 2620:0:861:ed1a::4 rapid count 100 source 2620:0:861:10d::253 PING6(56=40+8+8 bytes) 2620:0:861:10d::253 --> 2620:0:861:ed1a::4 Â !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cathal@nbgw:~$ mtr -b -w -z -c 10 2620:0:861:ed1a::4
Start: 2022-02-14T18:11:17+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6939 tunnel650354.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:32c::1) 10.0% 10 25.2 30.0 22.3 41.5 7.8
2. AS6939 e0-19.core2.lon2.he.net (2001:470:0:67::1) 10.0% 10 124.9 43.1 23.1 124.9 32.7
3. AS6939 100ge11-2.core1.lon2.he.net (2001:470:0:541::1) 0.0% 10 36.2 31.4 20.5 59.4 11.4
4. AS6939 100ge4-1.core1.nyc4.he.net (2001:470:0:2cf::2) 0.0% 10 92.6 97.4 85.0 147.7 18.6
5. AS6939 100ge11-1.core1.nyc5.he.net (2001:470:0:20a::2) 0.0% 10 93.9 98.6 85.1 142.3 16.6
6. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
7. AS6939 100ge1-2.core1.ash1.he.net (2001:470:0:277::1) 0.0% 10 94.1 109.7 93.2 193.9 30.8
8. AS6939 xe-5-3-3-500.cr1-eqiad.wikimedia.org (2001:470:0:1c0::2) 0.0% 10 116.2 100.1 90.1 130.0 12.8
9. AS14907 et-0-0-48-100.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:fe07::2) 0.0% 10 109.6 101.6 92.6 126.9 10.4
10. AS14907 2620:0:861:ed1a::4 0.0% 10 93.6 100.2 84.7 134.9 15.9
NOTE: This proves we can peer from the Anycast IP, but the switches will normally try to form the adjacency from their individual IP on the subnet. We can control this with "update source" on the switch if we ever need to, for the purpose of this test the only requirement was to confirm it will work if we need to do it.
BGP Peering on Vlan segment with Anycast GW, from Unicast IP
The purpose of this test is to validate that the QFX will allow a BGP adjacency to form to a device, connected via an access or trunk port, to its unique IRB interface IP on an int that has an anycast GW configured also.
Typically we want to peer directly with the GW. But if we ever run MC-LAG between two devices this would be problematic, two switches would be sharing this IP. Adjacency would form, but it would be random and only to a single device. So if we ever have that scenario we instead want to peer to each switch separately, using their unique IPs on that subnet rather than the shared VIP.
Relevant Config
The 3 scenarios are identical to in the previous set of tests (peering to the anycast VIP). Zero changes have been made on the QFX side, just the IP address configured as BGP peer has been adjusted on the test hosts in each case.
IPv4 Only
an-worker1147-test# show bgp ipv4 unicast neighbors 10.64.134.253 advertised-routes
BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
Default local pref 100, local AS 64605
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 208.80.154.228/32
0.0.0.0 0 32768Â ?
Route is accepted by the top of rack and public IP is pingable:
cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet.0
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 208.80.154.228/32 10.64.134.72 0 64605Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION rapid count 100 source 10.64.134.1 208.80.154.228 PING 208.80.154.228 (208.80.154.228): 56 data bytes  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
Start: 2022-02-15T17:31:07+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6830 176.61.34.1 0.0% 10 14.2 15.0 11.2 20.5 2.8
2. AS6830 109.255.255.254 0.0% 10 11.1 16.4 7.6 55.0 13.8
3. AS6830 ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42) 0.0% 10 10.1 15.9 10.1 30.4 6.0
4. AS6830 ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110) 0.0% 10 10.6 15.4 9.0 45.5 10.8
5. AS1299 dln-b2-link.ip.twelve99.net (62.115.172.136) 0.0% 10 19.0 13.3 7.7 20.4 4.2
6. AS1299 ldn-bb1-link.ip.twelve99.net (62.115.120.100) 0.0% 10 20.3 21.5 15.7 26.3 3.0
7. AS1299 nyk-bb2-link.ip.twelve99.net (62.115.113.20) 0.0% 10 94.1 97.3 94.4 100.2 2.4
8. AS1299 ash-bb2-link.ip.twelve99.net (62.115.136.201) 0.0% 10 99.2 101.2 97.1 105.5 3.1
9. AS1299 ash-b1-link.ip.twelve99.net (62.115.143.121) 0.0% 10 101.7 103.3 100.6 108.5 2.5
10. AS1299 wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226) 0.0% 10 101.4 99.7 95.6 103.2 2.4
11. AS14907 ae0.cr1-eqiad.wikimedia.org (208.80.154.193) 0.0% 10 105.5 103.1 97.0 110.5 5.0
12. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
13. AS14907 208.80.154.228 0.0% 10 101.4 99.4 94.8 104.6 3.0
IPv4 carrying IPv4 & IPv6 address families
Host was configured to send both types of route over the session:
an-worker1147-test# show bgp ipv4 neighbors 10.64.134.253 advertised-routes
BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
Default local pref 100, local AS 64600
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 208.80.154.228/32
0.0.0.0 0 32768Â ?
an-worker1147-test# show bgp ipv6 neighbors 10.64.134.253 advertised-routes
BGP table version is 3, local router ID is 208.80.154.228, vrf id 0
Default local pref 100, local AS 64600
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 2620:0:861:ed1a::4/128
 :: 0 32768 ?
IPv4 received and works:
cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet.0
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 208.80.154.228/32 10.64.134.72 0 64600Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 208.80.154.228 rapid count 100 source 10.64.134.253 PING 208.80.154.228 (208.80.154.228): 56 data bytes  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
Start: 2022-02-14T18:05:01+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6830 176.61.34.1 0.0% 10 12.5 15.0 10.3 26.0 4.8
2. AS6830 109.255.255.254 0.0% 10 17.2 13.3 9.8 17.2 2.4
3. AS6830 ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42) 0.0% 10 11.6 12.3 8.9 17.1 2.9
4. AS6830 ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110) 0.0% 10 28.3 13.4 9.0 28.3 5.7
5. AS1299 dln-b2-link.ip.twelve99.net (62.115.172.136) 0.0% 10 13.2 11.8 9.1 14.3 1.8
6. AS1299 ldn-bb1-link.ip.twelve99.net (62.115.120.100) 0.0% 10 22.4 23.3 19.0 33.0 3.7
7. AS1299 nyk-bb2-link.ip.twelve99.net (62.115.113.20) 0.0% 10 96.4 97.8 92.7 105.9 4.0
8. AS1299 ash-bb2-link.ip.twelve99.net (62.115.136.201) 0.0% 10 100.2 102.6 93.4 125.7 9.2
9. AS1299 ash-b1-link.ip.twelve99.net (62.115.143.121) 0.0% 10 98.5 102.6 97.1 106.7 3.1
10. AS1299 wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226) 0.0% 10 100.0 101.5 98.3 105.5 2.8
11. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
12. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
13. AS14907 208.80.154.228 0.0% 10 98.0 98.5 94.9 106.3 3.5
IPv6 also received and works:
cmooney@lsw1-e1-eqiad> show route receive-protocol bgp 10.64.134.72 table PRODUCTION.inet6.0
PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
2620:0:861:ed1a::4/128
* 2620:0:861:10d::72 0 64600Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION 2620:0:861:ed1a::4 rapid count 100 source 2620:0:861:10d::253 PING6(56=40+8+8 bytes) 2620:0:861:10d::253 --> 2620:0:861:ed1a::4 Â !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cathal@nbgw:~$ mtr -b -w -z -c 10 2620:0:861:ed1a::4
Start: 2022-02-14T17:39:50+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6939 tunnel650354.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:32c::1) 10.0% 10 23.9 28.5 22.1 43.7 7.8
2. AS6939 e0-19.core2.lon2.he.net (2001:470:0:67::1) 0.0% 10 35.8 41.3 23.8 110.6 28.7
3. AS6939 100ge11-2.core1.lon2.he.net (2001:470:0:541::1) 0.0% 10 24.7 29.0 20.1 61.5 14.2
4. AS6939 100ge4-1.core1.nyc4.he.net (2001:470:0:2cf::2) 0.0% 10 103.7 88.7 81.4 103.7 6.1
5. AS6939 100ge11-1.core1.nyc5.he.net (2001:470:0:20a::2) 0.0% 10 90.4 93.2 84.8 138.5 16.2
6. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
7. AS6939 100ge1-2.core1.ash1.he.net (2001:470:0:277::1) 0.0% 10 108.1 114.2 94.7 188.9 26.9
8. AS6939 xe-5-3-3-500.cr1-eqiad.wikimedia.org (2001:470:0:1c0::2) 0.0% 10 96.3 101.0 90.3 130.5 14.0
9. AS14907 et-0-0-48-100.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:fe07::2) 10.0% 10 94.6 101.7 92.6 155.8 20.3
10. AS14907 2620:0:861:ed1a::4 0.0% 10 110.5 103.3 89.6 140.9 16.9
IPv4 & IPv6 each carrying their own address family
IPv4 session up and route working:
an-worker1147-test# show bgp ipv4 neighbors 10.64.134.253 advertised-routes
BGP table version is 4, local router ID is 208.80.154.228, vrf id 0
Default local pref 100, local AS 64601
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 208.80.154.228/32
0.0.0.0 0 32768Â ?
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.64.134.72
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 208.80.154.228/32 10.64.134.72 0 64601Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 10.64.134.1 rapid count 100 208.80.154.228 PING 208.80.154.228 (208.80.154.228): 56 data bytes  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cathal@nbgw:~$ mtr -z -b -w -c 10 208.80.154.228
Start: 2022-02-15T17:19:44+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6830 176.61.34.1 0.0% 10 12.7 12.5 8.4 19.3 3.5
2. AS6830 109.255.255.254 0.0% 10 11.5 15.2 9.5 32.9 6.9
3. AS6830 ie-dub01a-rc1-ae-31-0.aorta.net (84.116.238.42) 0.0% 10 13.5 14.6 10.6 20.4 3.1
4. AS6830 ie-dub02a-ri1-ae-73-0.aorta.net (84.116.134.110) 0.0% 10 13.6 13.8 8.8 18.2 3.1
5. AS1299 dln-b2-link.ip.twelve99.net (62.115.172.136) 0.0% 10 11.3 13.7 10.3 17.7 2.6
6. AS1299 ldn-bb1-link.ip.twelve99.net (62.115.120.100) 0.0% 10 21.3 24.5 19.2 31.8 4.0
7. AS1299 nyk-bb2-link.ip.twelve99.net (62.115.113.20) 0.0% 10 99.5 96.3 92.8 101.2 2.8
8. AS1299 ash-bb2-link.ip.twelve99.net (62.115.136.201) 0.0% 10 100.0 100.6 97.6 106.0 2.7
9. AS1299 ash-b1-link.ip.twelve99.net (62.115.143.121) 0.0% 10 100.8 104.7 100.8 119.1 5.5
10. AS1299 wikimedia-ic308845-ash-b1.ip.twelve99-cust.net (80.239.132.226) 0.0% 10 99.8 102.0 96.5 109.1 3.9
11. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
12. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
13. AS14907 208.80.154.228 0.0% 10 104.1 98.3 94.5 104.1 3.1
IPv6 route ok also:
an-worker1147-test# show bgp ipv6 neighbors 2620:0:861:10d::253 advertised-routes
BGP table version is 3, local router ID is 208.80.154.228, vrf id 0
Default local pref 100, local AS 64601
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 2620:0:861:ed1a::4/128
 :: 0 32768 ?
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:10d::72
PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
2620:0:861:ed1a::4/128
* 2620:0:861:10d::72 0 64601Â ?
cmooney@lsw1-e1-eqiad> ping routing-instance PRODUCTION source 2620:0:861:10d::1 rapid count 100 2620:0:861:10d::72 PING6(56=40+8+8 bytes) 2620:0:861:10d::1 --> 2620:0:861:10d::72 Â !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cathal@nbgw:~$ mtr -b -w -z -c 10 2620:0:861:ed1a::4
Start: 2022-02-15T17:22:42+0000
HOST: nbgw Loss% Snt Last Avg Best Wrst StDev
1. AS6939 tunnel650354.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:32c::1) 0.0% 10 32.2 31.5 22.6 61.3 11.3
2. AS6939 e0-19.core2.lon2.he.net (2001:470:0:67::1) 0.0% 10 28.8 29.7 23.7 45.3 6.7
3. AS6939 100ge11-2.core1.lon2.he.net (2001:470:0:541::1) 0.0% 10 25.8 39.4 19.9 93.0 25.3
4. AS6939 100ge4-1.core1.nyc4.he.net (2001:470:0:2cf::2) 10.0% 10 86.2 94.6 86.2 140.8 17.5
5. AS6939 100ge11-1.core1.nyc5.he.net (2001:470:0:20a::2) 0.0% 10 92.0 97.1 86.9 127.6 15.7
6. AS??? Â ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
7. AS6939 100ge1-2.core1.ash1.he.net (2001:470:0:277::1) 10.0% 10 95.5 99.1 90.0 116.7 9.4
8. AS6939 xe-5-3-3-500.cr1-eqiad.wikimedia.org (2001:470:0:1c0::2) 0.0% 10 94.9 99.0 89.4 135.3 13.5
9. AS14907 et-0-0-48-100.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:fe07::2) 0.0% 10 93.3 104.5 92.2 145.2 19.3
10. AS14907 2620:0:861:ed1a::4 0.0% 10 98.0 98.6 89.9 126.4 12.3
BGP Re-establishment if device peered to Anycast GW moved to new switch (i.e. VM live motion)
This will need to be re-visited when we have a Ganeti cluster in the new cage. Without that it could not be tested.
The idea of this test is to confirm that a VM, with a BGP peering to its default gateway, will restore the BGP session in reasonable time if it's migrated to a new hypervisor host. As the new hypervisor will be connected to a different switch, the VM will get TCP RST back from the new top-of-rack when it sends TCP packets relating to the its BGP connection to the old switch. At this point it should re-initiate the session with the new top-of-rack and traffic should begin to flow again.
This was tested with vQFX lab devices and interruptions were less than 10 seconds, so we can be confident there won't be any issue.
eBGP Peering in VRF to external device
This test simulates doing an eBGP peering from the switch fabric overlay over a normal routed link. In production an example would be the connections from Spine switch to core routers. Assumption is it works similar to peering to a device on a connected Vlan from an IRB, but need to validate.
Relevant Config
Config in the test was to our CR routers in Eqiad. Separate peerings were established over IPv4 and IPv6 to exchange routes of each type:
cmooney@lsw1-e1-eqiad> show configuration interfaces et-0/0/48
description "Core: cr1-eqiad:et-1/0/2";
vlan-tagging;
mtu 9192;
encapsulation flexible-ethernet-services;
unit 100 {
description "cr1-eqiad et-1/0/2.100 - Production VRF";
vlan-id 100;
family inet {
address 10.66.0.9/31;
}
family inet6 {
address 2620:0:861:fe07::2/64;
}
}
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group EXTERNAL4
type external;
hold-time 30;
import DEFAULT4;
family inet {
unicast {
prefix-limit {
maximum 1000;
teardown;
}
}
}
export EXT_OUT4;
peer-as 14907;
neighbor 10.66.0.8 {
description cr1-eqiad;
}
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group EXTERNAL6
type external;
hold-time 30;
import DEFAULT6;
family inet6 {
unicast {
prefix-limit {
maximum 1000;
teardown;
}
}
}
export EXT_OUT6;
peer-as 14907;
neighbor 2620:0:861:fe07::1 {
description cr1-eqiad;
}
IPv4
IPv4 peering works and routes exchanged:
cmooney@lsw1-e1-eqiad> show bgp summary | match 14907 10.66.0.8 14907 106565 112060 0 6 1w4d 18:48:37 Establ 2620:0:861:fe07::1 14907 106564 112039 0 4 1w4d 18:48:26 Establ
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.66.0.8
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 0.0.0.0/0 10.66.0.8 14907 I
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 advertising-protocol bgp 10.66.0.8
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 10.64.130.0/24 Self I
* 10.64.131.0/24 Self I
* 10.64.132.0/24 Self I
* 10.64.134.0/24 Self I
* 10.64.138.0/24 Self I
* 10.64.142.0/24 Self I
* 208.80.154.226/32 Self 64600Â ?
* 208.80.154.228/32 Self 64601Â ?
IPv6
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:fe07::1
PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
*Â ::/0 2620:0:861:fe07::1 14907 I
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 advertising-protocol bgp 2620:0:861:fe07::1
PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 2620:0:861:100::/64 Self I
* 2620:0:861:109::/64 Self I
* 2620:0:861:10a::/64 Self I
* 2620:0:861:10b::/64 Self I
* 2620:0:861:10d::/64 Self I
* 2620:0:861:114::/64 Self I
2620:0:861:ed1a::3/128
* Self 64600Â ?
2620:0:861:ed1a::4/128
* Self 64601Â ?
2620:0:861:fe07::/64
* Self I
2620:0:861:fe08::/64
* Self I
BGP Route propagaton from unicast peer into EVPN and into remote VRF table
This test validates routes learnt from unicast BGP peers in the overlay are properly propagated as an EVPN type 5 routes between devices in the switch fabric, and pushed to the correct routing table/VRF on remote devices based on the parameters in the EVPN route.
Routes learnt on Spine1:
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.66.0.8
PRODUCTION.inet.0: 35 destinations, 38 routes (35 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 0.0.0.0/0 10.66.0.8 14907 I
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:fe07::1
PRODUCTION.inet6.0: 41 destinations, 45 routes (41 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
*Â ::/0 2620:0:861:fe07::1 14907 I
Routes learnt on Spine2:
cmooney@lsw1-f1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.66.0.10
PRODUCTION.inet.0: 32 destinations, 35 routes (32 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 0.0.0.0/0 10.66.0.10 14907 I
cmooney@lsw1-f1-eqiad> show route table PRODUCTION.inet6.0 receive-protocol bgp 2620:0:861:fe08::1
PRODUCTION.inet6.0: 38 destinations, 42 routes (38 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
*Â ::/0 2620:0:861:fe08::1 14907 I
These are exported as EVPN type 5 routes, announced and received by downstream LEAF switch lsw1-e3-eqiad:
cmooney@lsw1-e3-eqiad> show route table bgp.evpn.0 aspath-regex ".* 14907$"
bgp.evpn.0: 79 destinations, 147 routes (79 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
5:10.64.128.3:5000::0::0.0.0.0::0/248
*[BGP/170] 1w4d 19:04:01, localpref 100, from 10.64.128.3
AS path: 14907 I, validation-state: unverified
> to 10.64.129.2 via et-0/0/54.0
[BGP/170] 1w4d 19:04:00, localpref 100, from 10.64.128.7
AS path: 14907 I, validation-state: unverified
> to 10.64.129.2 via et-0/0/54.0
5:10.64.128.7:5000::0::0.0.0.0::0/248
*[BGP/170] 1w4d 23:37:43, localpref 100, from 10.64.128.7
AS path: 14907 I, validation-state: unverified
> to 10.64.129.16 via et-0/0/55.0
[BGP/170] 1w4d 23:37:43, localpref 100, from 10.64.128.3
AS path: 14907 I, validation-state: unverified
> to 10.64.129.16 via et-0/0/55.0
5:10.64.128.3:5000::0::::::0/248
*[BGP/170] 1w4d 19:03:49, localpref 100, from 10.64.128.3
AS path: 14907 I, validation-state: unverified
> to 10.64.129.2 via et-0/0/54.0
[BGP/170] 1w4d 19:03:49, localpref 100, from 10.64.128.7
AS path: 14907 I, validation-state: unverified
> to 10.64.129.2 via et-0/0/54.0
5:10.64.128.7:5000::0::::::0/248
*[BGP/170] 1w4d 23:37:39, localpref 100, from 10.64.128.7
AS path: 14907 I, validation-state: unverified
> to 10.64.129.16 via et-0/0/55.0
[BGP/170] 1w4d 23:37:39, localpref 100, from 10.64.128.3
AS path: 14907 I, validation-state: unverified
> to 10.64.129.16 via et-0/0/55.0
These are imported into the local EVPN table as shown here:
cmooney@lsw1-e3-eqiad> show evpn ip-prefix-database prefix 0.0.0.0/0
L3 context: PRODUCTION
EVPN->IPv4 Imported Prefixes
Prefix Etag
0.0.0.0/0 0
Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI
10.64.128.3:5000 3005000 a4:e1:1a:81:3a:80 10.64.128.3
10.64.128.7:5000 3005000 a4:e1:1a:81:9e:80 10.64.128.7
cmooney@lsw1-e3-eqiad> show evpn ip-prefix-database prefix ::/0
L3 context: PRODUCTION
EVPN->IPv6 Imported Prefixes
Prefix Etag
 ::/0 0
Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI
10.64.128.3:5000 3005000 a4:e1:1a:81:3a:80 10.64.128.3
10.64.128.7:5000 3005000 a4:e1:1a:81:9e:80 10.64.128.7
The IPv4 route is imported into the routing table for the 'PRODUCTION' routing-instance, which is what we expect based on the configuration:
cmooney@lsw1-e3-eqiad> show route table PRODUCTION.inet.0 0.0.0.0/0 exact
PRODUCTION.inet.0: 30 destinations, 36 routes (30 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both
0.0.0.0/0 @[EVPN/170] 1w4d 22:56:36
> to 10.64.129.2 via et-0/0/54.0
[EVPN/170] 1w5d 03:30:18
> to 10.64.129.16 via et-0/0/55.0
#[Multipath/255] 1w4d 22:56:36, metric2 8
> to 10.64.129.2 via et-0/0/54.0
to 10.64.129.16 via et-0/0/55.0
This is also true for IPv6:
cmooney@lsw1-f2-eqiad> show route table PRODUCTION.inet6.0Â ::/0 exact
PRODUCTION.inet6.0: 37 destinations, 44 routes (37 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both
 ::/0 @[EVPN/170] 1w5d 22:23:30
> to 10.64.129.8 via et-0/0/54.0
[EVPN/170] 1w5d 19:39:18
> to 10.64.129.22 via et-0/0/55.0
#[Multipath/255] 1w5d 19:39:18, metric2 8
> to 10.64.129.8 via et-0/0/54.0
to 10.64.129.22 via et-0/0/55.0
BGP Route propagation to external hosts from VRF [IPv4]
The purpose of this test is to show that routes originating in the EVPN overlay / VRF are properly advertised to external peers.
The check is mainly that those routes are announced from the Spine switches to our CRs correctly.
IPv4
Relevant Config
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group EXTERNAL4
type external;
hold-time 30;
import DEFAULT4;
family inet {
unicast {
prefix-limit {
maximum 1000;
teardown;
}
}
}
export EXT_OUT4;
peer-as 14907;
neighbor 10.66.0.8 {
description cr1-eqiad;
}
cmooney@lsw1-e1-eqiad> show configuration policy-options policy-statement EXT_OUT4
/* Controls export of routes from VRF to external peers */
term LVS {
from {
protocol [ bgp evpn ];
as-path LOCAL_LVS;
}
then accept;
}
term K8_ANYCAST {
from {
protocol [ bgp evpn ];
as-path LOCAL_K8_ANYCAST;
}
then accept;
}
term NETWORKS {
from {
protocol [ direct evpn ];
route-filter 0.0.0.0/0 prefix-length-range /0-/29;
}
then accept;
}
then reject;
Results:
Top-of-rack switch subnets are propagated as expected (and, somewhat importantly, /32 host routes from EVPN type 2 sources / ARP snooping are not announced):
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 advertising-protocol bgp 10.66.0.8
PRODUCTION.inet.0: 30 destinations, 31 routes (30 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 10.64.130.0/24 Self I
* 10.64.131.0/24 Self I
* 10.64.132.0/24 Self I
* 10.64.134.0/24 Self I
* 10.64.138.0/24 Self I
* 10.64.142.0/24 Self I
* 208.80.154.226/32 Self 64600Â ?
* 208.80.154.229/32 Self 64600Â ?
IPv6
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group EXTERNAL6
type external;
hold-time 30;
import DEFAULT6;
family inet6 {
unicast {
prefix-limit {
maximum 1000;
teardown;
}
}
}
export EXT_OUT6;
peer-as 14907;
neighbor 2620:0:861:fe07::1 {
description cr1-eqiad;
}
cmooney@lsw1-e1-eqiad> show configuration policy-options policy-statement EXT_OUT6
/* Controls export of routes from VRF to external peers. */
term LVS {
from {
protocol [ bgp evpn ];
as-path LOCAL_LVS;
}
then accept;
}
term EVPN_K8_ANYCAST {
from {
protocol [ bgp evpn ];
as-path LOCAL_K8_ANYCAST;
}
then accept;
}
term NETWORKS {
from {
protocol [ direct evpn ];
route-filter ::/0 prefix-length-range /0-/125;
}
then accept;
}
then reject;
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet6.0 advertising-protocol bgp 2620:0:861:fe07::1
PRODUCTION.inet6.0: 36 destinations, 37 routes (36 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 2620:0:861:100::/64 Self I
* 2620:0:861:109::/64 Self I
* 2620:0:861:10a::/64 Self I
* 2620:0:861:10b::/64 Self I
* 2620:0:861:114::/64 Self I
2620:0:861:ed1a::3/128
* Self 64600Â ?
2620:0:861:ed1a::6/128
* Self 64600Â ?
2620:0:861:fe07::/64
* Self I
2620:0:861:fe08::/64
* Self I
BGP Peering to external host from VRF using local-as 14907
If possible we want to avoid having to configure end-hosts with the ASN of the top-of-rack switch they are connected to. Instead our preference is to "fake" the AS used for peering from the switch side, always using the WMF public AS 14907, so that this piece of the configuration can be kept constant on the server side.
In order to do this the switch needs to support some form of "local-as" override. We need to be careful that:
- BGP session establishes.
- AS14907 is not placed in the AS-path when prefixes are propagated onwards (i.e. sent to CRs).
- The device using fake 'local-as' 14907 can still establish and learn routes from an eBGP peer using that ASN (i.e. learnt from CRs).
- If announcing routes over the BGP session only the the configured 'local-as' is included in the AS-path, the switch's default ASN is not present.
Relevant Config
The previous configuration for the PyBal group was modified with the 'local-as' configuration as shown:
cmooney@lsw1-e1-eqiad> show configuration routing-instances PRODUCTION protocols bgp group PyBal
type external;
hold-time 30;
import LVS_IMPORT;
family inet {
unicast {
prefix-limit {
maximum 1000;
teardown 20;
}
}
}
family inet6 {
unicast {
prefix-limit {
maximum 1000;
teardown 20;
}
}
}
export NONE;
peer-as 64600;
local-as 14907 loops 2 private no-prepend-global-as;
neighbor 10.64.130.10 {
description ms-fe1012-test;
}
To break down the meaning of these commands:
local-as: Determines what AS to use for session loops 2: With 'local-as' set to 14907 the switch will not accept that AS in a received path, "loops 2" allows routes with 1 instance of it in. private: This tells JunOS the 'local-as' is only relevant to the peering, it should use its normal ASN when propagating routes learnt on it. no-prepend-global-as: This tells JunOS to not put its own, globally-configured ASN, into the as-path if announcing routes to this peer.
Results
Adjacency forms from test-host with peer-as set to 14907:
ms-fe1012-test# show bgp ipv4 unicast summary BGP router identifier 208.80.154.226, local AS number 64600 vrf-id 0 BGP table version 19 RIB entries 3, using 552 bytes of memory Peers 1, using 723 KiB of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc 10.64.130.1 4 14907 3256 2968 0 0 0 06:46:07 0 1 N/A
The route being announced is accepted and learnt with as-path of the "PyBal" peer (64600):
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.64.130.10
PRODUCTION.inet.0: 30 destinations, 31 routes (30 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 208.80.154.226/32 10.64.130.10 0 64600Â ?
This is correctly being advertised externally to the CR, which we are peering to on its AS 14907:
cmooney@lsw1-e1-eqiad> show bgp summary | match 14907 10.66.0.8 14907 2147 2266 0 0 5:41:29 Establ
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 advertising-protocol bgp 10.66.0.8 | match 208.80.154.226 * 208.80.154.226/32 Self 64600Â ?
On the CR the route is learnt with the switch's ASN followed by the PyBal one, as desired:
cmooney@re0.cr1-eqiad> show route receive-protocol bgp 10.66.0.9 | match 208.80.154.226 * 208.80.154.226/32 10.66.0.9 64810 64600Â ?
On the switch we still learn and accept the route sent by the CR (with AS14907 in the path):
cmooney@lsw1-e1-eqiad> show route table PRODUCTION.inet.0 receive-protocol bgp 10.66.0.8
PRODUCTION.inet.0: 30 destinations, 31 routes (30 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 0.0.0.0/0 10.66.0.8 14907 I
Inernal routing from device connected to access vlan
This tests validates connectivity between hosts in the new racks and devices connected to VC-based switches in other rows.
IPv4
root@elastic1093-test:~# mtr --address 10.64.132.2 -b -w -c 10 10.64.16.8
Start: 2022-02-17T21:13:05+0000
HOST: elastic1093-test Loss% Snt Last Avg Best Wrst StDev
1.|-- irb-1033.lsw1-e3-eqiad.eqiad.wmnet (10.64.132.1) 0.0% 10 4.4 4.7 2.0 8.4 2.4
2.|-- irb-1035.lsw1-f1-eqiad.eqiad.wmnet (10.64.134.1) 0.0% 10 1.3 3.4 0.9 6.2 1.9
3.|-- et-1-0-2-100.cr2-eqiad.eqiad.wmnet (10.66.0.10) 0.0% 10 0.3 1.0 0.3 6.1 1.8
4.|-- phab1001.eqiad.wmnet (10.64.16.8) 0.0% 10 0.2 0.3 0.2 0.3 0.0
IPv6
root@elastic1093-test:~# mtr --address 2620:0:861:10b:10:64:132:2 -b -w -c 10 2620:0:861:102:10:64:16:8
Start: 2022-02-17T21:12:33+0000
HOST: elastic1093-test Loss% Snt Last Avg Best Wrst StDev
1.|-- irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1) 0.0% 10 0.6 0.5 0.5 0.6 0.0
2.|-- irb-1039.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:100::1) 0.0% 10 0.6 0.6 0.5 0.7 0.1
3.|-- et-1-0-2-100.cr1-eqiad.eqiad.wmnet (2620:0:861:fe07::1) 0.0% 10 0.4 0.6 0.4 1.8 0.4
4.|-- phab1001.eqiad.wmnet (2620:0:861:102:10:64:16:8) 0.0% 10 0.3 0.3 0.3 0.3 0.0
External routing from device connected to access vlan
This tests validates connectivity between hosts in the new racks and devices connected to VC-based switches in other rows.
IPv4
root@elastic1093-test:~# mtr -4 --address 208.80.154.229 -z -b -w -c 10 www.ietf.org
Start: 2022-02-17T21:16:38+0000
HOST: elastic1093-test Loss% Snt Last Avg Best Wrst StDev
1. AS??? irb-1033.lsw1-e3-eqiad.eqiad.wmnet (10.64.132.1) 0.0% 10 7.1 5.2 0.7 13.3 3.9
2. AS??? irb-1035.lsw1-f1-eqiad.eqiad.wmnet (10.64.134.1) 0.0% 10 1.8 4.5 1.4 7.8 2.3
3. AS??? et-1-0-2-100.cr2-eqiad.eqiad.wmnet (10.66.0.10) 0.0% 10 0.4 0.5 0.4 0.6 0.1
4. AS??? 13335.ash.equinix.com (206.126.237.30) 60.0% 10 1.8 7.7 1.8 13.3 6.4
5. AS13335 172.70.172.2 0.0% 10 1.7 3.0 1.2 11.5 3.2
6. AS13335 104.16.45.99 0.0% 10 1.2 1.2 1.1 1.3 0.0
IPv6
root@elastic1093-test:~# mtr -6 --address 2620:0:861:ed1a::6 -z -b -w -c 10 www.ietf.org
Start: 2022-02-17T21:17:12+0000
HOST: elastic1093-test Loss% Snt Last Avg Best Wrst StDev
1. AS14907 irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1) 0.0% 10 0.6 0.5 0.4 0.6 0.1
2. AS14907 irb-1039.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:100::1) 0.0% 10 7.9 5.2 0.5 40.2 12.5
3. AS14907 et-1-0-2-100.cr1-eqiad.eqiad.wmnet (2620:0:861:fe07::1) 0.0% 10 0.6 2.4 0.4 18.9 5.8
4. AS14907 ae0.cr2-eqiad.wikimedia.org (2620:0:861:fe00::2) 0.0% 10 0.3 0.4 0.3 0.5 0.0
5. AS??? 13335.ash.equinix.com (2001:504:0:2:0:1:3335:1) 60.0% 10 1.3 5.4 1.3 17.0 7.8
6. AS13335 2400:cb00:350:3:: 0.0% 10 1.3 4.2 1.1 15.2 5.3
7. AS13335 2606:4700::6810:2c63 0.0% 10 0.4 0.5 0.4 0.8 0.1
Worth observing that hop 2 is different in each case, as lsw1-e3 has two upstream "spines" (lsw1-e1 and lsw1-f1), and load-balances traffic across them.
ARP Supression
ARP suppression is a feature in EVPN layer-2 networks, which aims to minimise the amount of layer-2 broadcast traffic being sent, and thus support larger layer-2 segments. It is made possible through the inclusion of IP as well as MAC information in EVPN type 2 routes. The IP data is inserted by switches based on either on local ARP tables or snooping on traffic. As this is distributed in BGP/EVPN, devices become aware of remote MAC/IP bindings, and can answer ARP requests from their own hosts if they have the answer.
Checks
Ping Anycast GW 10.64.134.1 of lsw1-e1 from directly connected 10.64.131.72.
root@an-worker1147-test:~# ping -I 10.64.134.72 -c 2 10.64.134.1 PING 10.64.134.1 (10.64.134.1) from 10.64.134.72Â : 56(84) bytes of data. 64 bytes from 10.64.134.1: icmp_seq=1 ttl=64 time=5.28 ms 64 bytes from 10.64.134.1: icmp_seq=2 ttl=64 time=2.71 ms
ARP entry is populated on lsw1-e1 as desired:
cmooney@lsw1-e1-eqiad> show arp no-resolve expiration-time | match 10.64.134.72 e4:3d:1a:54:14:45 10.64.134.72 irb.1035 [xe-0/0/6.0] permanent remote
In turn an EVPN route is learnt on lsw1-f1 containing this information:
cmooney@lsw1-f1-eqiad> show route protocol bgp table bgp.evpn.0 terse | match 10.64.134.72
2:10.64.128.3:64810::2001035::e4:3d:1a:54:14:45::10.64.134.72/304 MAC/IP
EVPN database shows it in more detail:
cmooney@lsw1-e1-eqiad> show evpn database extensive mac-address e4:3d:1a:54:14:45
Instance: default-switch
VN Identifier: 2001035, MAC address: e4:3d:1a:54:14:45
State: 0x0
Source: xe-0/0/6.0, Rank: 1, Status: Active
Mobility sequence number: 0 (minimum origin address 10.64.128.3)
Timestamp: Feb 17 21:38:12.700143 (0x620ec044)
State: <Local-MAC-Only Local-To-Remote-Adv-Allowed>
MAC advertisement route status: Created
IP address: 10.64.134.72
Flags: <Local-Adv>
L3 route: 10.64.134.72/32, L3 context: PRODUCTION (irb.1035)
IP address: 2620:0:861:10d::72
Flags: <Local-Adv>
L3 route: 2620:0:861:10d::72/128, L3 context: PRODUCTION (irb.1035)
IP address: fe80::e63d:1aff:fe54:1445
Flags: <Local-Adv>
L3 route: fe80::e63d:1aff:fe54:1445/128, L3 context: PRODUCTION (irb.1035)
History db:
Time Event
Feb 17 12:36:30.139 2022 xe-0/0/6.0Â : Created
Feb 17 12:36:30.139 2022 Updating output state (change flags 0x1 <ESI-Added>)
Feb 17 12:36:30.139 2022 Active ESI changing (not assigned -> xe-0/0/6.0)
Feb 17 12:36:30.139 2022 xe-0/0/6.0Â : Updating output state (change flags 0x200 <IP-Added>)
Feb 17 12:37:40.161 2022 xe-0/0/6.0Â : Updating output state (change flags 0x200 <IP-Added>)
Feb 17 21:35:36.597 2022 xe-0/0/6.0Â : 10.64.134.72 Selected IRB interface nexthop
Feb 17 21:35:36.597 2022 xe-0/0/6.0Â : 2620:0:861:10d::72 Selected IRB interface nexthop
Feb 17 21:38:12.700 2022 xe-0/0/6.0Â : Updating output state (change flags 0x200 <IP-Added>)
Feb 17 21:38:12.700 2022 xe-0/0/6.0Â : fe80::e63d:1aff:fe54:1445 Selected IRB interface nexthop
Ultimately this causes an ARP entry to be added on the lsw1-f1 binding the same MAC/IP:
cmooney@lsw1-f1-eqiad> show arp no-resolve expiration-time | match 10.64.134.72 e4:3d:1a:54:14:45 10.64.134.72 irb.1035 [vtep.32769] permanent remote
Test
Now that the IP/MAC info for 10.64.134.72 is populated across the network we need to check what happens when we send an ARP request for this IP from a host on the same Vlan, but connected to a different switch.
To begin I connected a host port with no IP address configured to a port on lsw1-f1, port was configured as an access port in Vlan 1035 / private1-f1-eqiad. Running 'date' a lot so the timing is clearer, I first added an IP on the hosts interface, making this the first moment it could participate in ARP:
root@an-worker1148-test:/etc/systemd# date && ip addr add 10.64.134.11/24 dev eno2np1 Thu 17 Feb 22:54:59 GMT 2022
I then pinged the Anycast GW IP (which will be lsw1-f1 as that's what we're connected to):
root@an-worker1148-test:/etc/systemd# date && ping 10.64.134.1 Thu 17 Feb 22:55:29 GMT 2022 PING 10.64.134.1 (10.64.134.1) 56(84) bytes of data. 64 bytes from 10.64.134.1: icmp_seq=1 ttl=64 time=98.0 ms 64 bytes from 10.64.134.1: icmp_seq=2 ttl=64 time=1.14 ms
And now ping for the other host, connected to remote switch lsw1-e1:
root@an-worker1148-test:/etc/systemd# date && ping 10.64.134.72 Thu 17 Feb 22:55:33 GMT 2022 PING 10.64.134.72 (10.64.134.72) 56(84) bytes of data. 64 bytes from 10.64.134.72: icmp_seq=1 ttl=64 time=14.5 ms 64 bytes from 10.64.134.72: icmp_seq=2 ttl=64 time=0.230 ms
Looking over packet captures running on this device, it sent ARPs as expected for both IPs pinged, and got answers for both:
root@an-worker1148-test:~# date && tcpdump -i eno2np1 -l -nn arp && date Thu 17 Feb 22:54:58 GMT 2022 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eno2np1, link-type EN10MB (Ethernet), snapshot length 262144 bytes 22:55:29.065596 ARP, Request who-has 10.64.134.1 tell 10.64.134.11, length 28 22:55:29.075964 ARP, Reply 10.64.134.1 is-at 00:00:5e:11:fa:ce, length 46 22:55:33.324530 ARP, Request who-has 10.64.134.72 tell 10.64.134.11, length 28 22:55:33.338783 ARP, Reply 10.64.134.72 is-at e4:3d:1a:54:14:45, length 46
A tcpdump on 10.64.134.72 reveals, however, that no ARP request hit the server from 10.64.134.11. The first packets we see from the other IP are the ICMP packets:
root@an-worker1147-test:/etc/systemd# date && tcpdump -i eno2np1.1035 -l -nn host 10.64.134.11 or arp && date Thu 17 Feb 22:54:54 GMT 2022 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eno2np1.1035, link-type EN10MB (Ethernet), snapshot length 262144 bytes 22:55:33.338005 IP 10.64.134.11 > 10.64.134.72: ICMP echo request, id 16418, seq 1, length 64 22:55:33.338049 IP 10.64.134.72 > 10.64.134.11: ICMP echo reply, id 16418, seq 1, length 64 22:55:34.325397 IP 10.64.134.11 > 10.64.134.72: ICMP echo request, id 16418, seq 2, length 64 22:55:34.325424 IP 10.64.134.72 > 10.64.134.11: ICMP echo reply, id 16418, seq 2, length 64
We can thus conclude the ARP request an-worker1147 sent from 10.64.134.71 was not broadcast to all devices across the network, and instead lsw1-f1 generated the appropriate response when it received it, using the information from EVPN. This is an adequate, if not a very sophisticated, way to verify this. Unfortunately the QFX doesn't give any stats or counters on it, and Juniper's own test instructions are similarly based on doing packet captures.
ND Suppression
ND suppression in IPv6 is very similar to the concept of ARP suppression in IPv4. Once one switch has the IP/MAC binding it's included in EVPN type 2 routes, which mean the whole network should have this info. If other hosts perform neighbor discovery for a given IP, the packets are not multicast in the Vlan but the switches should generate a fake response, with the correct info learnt from EVPN, reducing the level of BUM traffic.
Configuration / vlan is the same as before.
Results
Again 'date' used to make the sequence clearer. First the IP address is added to the interface:
root@an-worker1148-test:~# date && ip addr add 2620:0:861:10d::11/64 dev eno2np1 Thu 17 Feb 23:34:12 GMT 2022
Then we ping the Anycast GW (will hit lsw1-f1 where we're connected):
root@an-worker1148-test:~# date && ping 2620:0:861:10d::1 Thu 17 Feb 23:34:22 GMT 2022 PING 2620:0:861:10d::1(2620:0:861:10d::1) 56 data bytes 64 bytes from 2620:0:861:10d::1: icmp_seq=1 ttl=64 time=12.2 ms 64 bytes from 2620:0:861:10d::1: icmp_seq=2 ttl=64 time=0.491 ms
Lastly we ping an-worker1147, connected in the same Vlan on a remote switch:
root@an-worker1148-test:~# date && ping 2620:0:861:10d::72 Thu 17 Feb 23:34:29 GMT 2022 PING 2620:0:861:10d::72(2620:0:861:10d::72) 56 data bytes 64 bytes from 2620:0:861:10d::72: icmp_seq=1 ttl=64 time=21.2 ms 64 bytes from 2620:0:861:10d::72: icmp_seq=2 ttl=64 time=0.232 ms
A packet capture on the device doing the pings shows several ND requests, including for an-worker1148:
root@an-worker1148-test:~# date && tcpdump -i eno2np1 -l -nn icmp6 Thu 17 Feb 23:33:40 GMT 2022 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eno2np1, link-type EN10MB (Ethernet), snapshot length 262144 bytes 23:34:12.918598 IP6Â :: > ff02::1:ff00:11: ICMP6, neighbor solicitation, who has 2620:0:861:10d::11, length 32 23:34:22.706295 IP6 2620:0:861:10d::11 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2620:0:861:10d::1, length 32 23:34:22.710987 IP6 2620:0:861:10d::1 > 2620:0:861:10d::11: ICMP6, neighbor advertisement, tgt is 2620:0:861:10d::1, length 32 23:34:22.711020 IP6 2620:0:861:10d::11 > 2620:0:861:10d::1: ICMP6, echo request, id 10401, seq 1, length 64 23:34:22.718416 IP6 2620:0:861:10d::1 > 2620:0:861:10d::11: ICMP6, echo reply, id 10401, seq 1, length 64 23:34:23.707671 IP6 2620:0:861:10d::11 > 2620:0:861:10d::1: ICMP6, echo request, id 10401, seq 2, length 64 23:34:23.708132 IP6 2620:0:861:10d::1 > 2620:0:861:10d::11: ICMP6, echo reply, id 10401, seq 2, length 64 23:34:29.649454 IP6 2620:0:861:10d::11 > ff02::1:ff00:72: ICMP6, neighbor solicitation, who has 2620:0:861:10d::72, length 32 23:34:29.659961 IP6 2620:0:861:10d::72 > 2620:0:861:10d::11: ICMP6, neighbor advertisement, tgt is 2620:0:861:10d::72, length 32 23:34:29.659993 IP6 2620:0:861:10d::11 > 2620:0:861:10d::72: ICMP6, echo request, id 38872, seq 1, length 64 23:34:29.670614 IP6 2620:0:861:10d::72 > 2620:0:861:10d::11: ICMP6, echo reply, id 38872, seq 1, length 64 23:34:30.650792 IP6 2620:0:861:10d::11 > 2620:0:861:10d::72: ICMP6, echo request, id 38872, seq 2, length 64 23:34:30.650994 IP6 2620:0:861:10d::72 > 2620:0:861:10d::11: ICMP6, echo reply, id 38872, seq 2, length 64
Capture on an-worker1148 shows, however, that the ND message never hit it:
root@an-worker1147-test:~# tcpdump -i eno2np1.1035 -l -nn icmp6 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eno2np1.1035, link-type EN10MB (Ethernet), snapshot length 262144 bytes 23:34:29.656794 IP6 2620:0:861:10d::11 > 2620:0:861:10d::72: ICMP6, echo request, id 38872, seq 1, length 64 23:34:29.667235 IP6 2620:0:861:10d::72 > 2620:0:861:10d::11: ICMP6, echo reply, id 38872, seq 1, length 64 23:34:30.647585 IP6 2620:0:861:10d::11 > 2620:0:861:10d::72: ICMP6, echo request, id 38872, seq 2, length 64 23:34:30.647613 IP6 2620:0:861:10d::72 > 2620:0:861:10d::11: ICMP6, echo reply, id 38872, seq 2, length 64
Similar to the ARP test this shows that the switches are generating responses to ND requests locally, reducing BUM traffic in the Vlan.
DHCP Relay & Option 82 insertion
We format DHCP requests from end servers over unicast IPv4 to our 'install' servers in each datacenter. Those machines run ISC's dhcpd software and return IP addresses to end hosts if configured to do so for the specific end host. Determination of which host is requesting is based on the "option 82" information present in the DHCP DISCOVER messages from the host. This information is inserted by the top-of-rack switch when the packet goes through it, allowing this to take place.
Relevant Config
In our case the DHCP requests will arrive from IRB interface in an overlay VRF/routing-instance, and be relayed to the install server via the same instance. So the configuration is done in that part:
cmooney@lsw1-e3-eqiad> show configuration routing-instances PRODUCTION forwarding-options
dhcp-relay {
relay-option-82 {
circuit-id {
prefix {
host-name;
}
}
}
server-group {
INSTALL-SERVER {
208.80.154.32;
}
}
group DHCP-RELAY {
active-server-group INSTALL-SERVER;
interface irb.1033;
interface irb.1041;
}
}
Results
We can then send a DHCP request from a connected host, for instance elastic1093-test connected to lsw1-e3 xe-0/0/21.
DHCP DISCOVER is received on install1003 shortly after, with option 82 information populated as expected:
cmooney@install1003:~$ sudo tcpdump -i ens5 -l -p -vvv -nn port 67 or port 68 and not src host 208.80.154.199
tcpdump: listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes
23:56:56.244662 IP (tos 0x0, ttl 62, id 6901, offset 0, flags [none], proto UDP (17), length 408)
10.64.132.1.67 > 208.80.154.32.67: [udp sum ok] BOOTP/DHCP, Request from e4:3d:1a:cb:5c:21, length 380, hops 1, xid 0xc24c6629, Flags [none] (0x0000)
Gateway-IP 10.64.132.1
Client-Ethernet-Address e4:3d:1a:cb:5c:21
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Hostname Option 12, length 16: "elastic1093-test"
Parameter-Request Option 55, length 13:
Subnet-Mask, BR, Time-Zone, Default-Gateway
Domain-Name, Domain-Name-Server, Option 119, Hostname
Netbios-Name-Server, Netbios-Scope, MTU, Classless-Static-Route
NTP
Agent-Information Option 82, length 78:
Circuit-ID SubOption 1, length 43: lsw1-e3-eqiad:xe-0/0/21.0:private1-e3-eqiad
Unknown SubOption 12, length 31:
0x0000: 0002 0000 0000 0583 0100 0000 6134 3a65
0x0010: 313a 3161 3a38 313a 6435 3a38 3000 00
END Option 255, length 0
PAD Option 0, length 0, occurs 23
IPv6 Router Advertisement Generation
Although we statically configure end-host IPs in /etc/network/interfaces, the IPv6 address that is added there needs to be derived on a given host from a SLAAC-assigned address (giving us the subnet) and the IPv4 address (which we use as host portion of v6 address).
IPv6 router advertisements must be generated on all IRB/Vlan interfaces and sent to end hosts so they can configure an IPv6 global unicast address via SLAAC.
Relevant Config
cmooney@lsw1-e3-eqiad> show configuration interfaces irb.1033
description private1-e3-eqiad;
family inet {
address 10.64.132.1/24;
}
family inet6 {
address 2620:0:861:10b::1/64;
}
cmooney@lsw1-e3-eqiad> show configuration protocols router-advertisement
interface irb.1033 {
max-advertisement-interval 30;
default-lifetime 600;
prefix 2620:0:861:10b::1/64;
}
Results
RAs are received on end hosts connected to Vlan1033 as expected:
cmooney@elastic1093:~$ sudo tcpdump -e -v -l -nn -i enp59s0f0np0 icmp6 tcpdump: listening on enp59s0f0np0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 13:28:55.906194 a4:e1:1a:81:d5:80 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 110: (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::a6e1:1a04:981:d580 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 56 hop limit 64, Flags [none], pref medium, router lifetime 600s, reachable time 0ms, retrans timer 0ms source link-address option (1), length 8 (1): a4:e1:1a:81:d5:80 prefix info option (3), length 32 (4): 2620:0:861:10b::/64, Flags [onlink, auto], valid time 2592000s, pref. time 604800s
DHCP Relay On Stretched Vlan / IRB with Anycast GW
Similar to in the previous case, we send a DHCP request from an-worker1148, connected to lsw1-f1 on port xe-0/0/6.
IRB is configured with Anycast GW config like this:
cmooney@lsw1-f1-eqiad> show configuration interfaces irb.1035
virtual-gateway-accept-data;
description private1-f1-eqiad;
family inet {
address 10.64.134.254/24 {
preferred;
virtual-gateway-address 10.64.134.1;
}
}
family inet6 {
address 2620:0:861:10d::254/64 {
preferred;
virtual-gateway-address 2620:0:861:10d::1;
}
}
virtual-gateway-v4-mac 00:00:5e:11:fa:ce;
virtual-gateway-v6-mac 00:00:5e:11:fa:ce;
When we run the DHCP client on an-worker1148 we see this on the install server. Worth noting in the source IP is lsw1-e1-eqiad's unique IP on the subnet, rather than the VIP shared by all devices (this is important to ensure replies go back to correct device):
cmooney@install1003:~$ sudo tcpdump -i ens5 -l -p -vvv -nn net 10.64.134.0/24
tcpdump: listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes
00:09:47.764115 IP (tos 0x0, ttl 63, id 23228, offset 0, flags [none], proto UDP (17), length 407)
10.64.134.254.67 > 208.80.154.32.67: [udp sum ok] BOOTP/DHCP, Request from e4:3d:1a:54:ab:a7, length 379, hops 1, xid 0x8bee627f, Flags [none] (0x0000)
Gateway-IP 10.64.134.254
Client-Ethernet-Address e4:3d:1a:54:ab:a7
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Hostname Option 12, length 18: "an-worker1148-test"
Parameter-Request Option 55, length 13:
Subnet-Mask, BR, Time-Zone, Default-Gateway
Domain-Name, Domain-Name-Server, Option 119, Hostname
Netbios-Name-Server, Netbios-Scope, MTU, Classless-Static-Route
NTP
Agent-Information Option 82, length 77:
Circuit-ID SubOption 1, length 42: lsw1-f1-eqiad:xe-0/0/6.0:private1-f1-eqiad
Unknown SubOption 12, length 31:
0x0000: 0002 0000 0000 0583 0100 0000 6134 3a65
0x0010: 313a 3161 3a38 313a 3965 3a38 3000 00
END Option 255, length 0
PAD Option 0, length 0, occurs 21
NOTE: In addition to the above packet, the install server also received a relayed DHCP request from lsw1-e1-eqiad, the only other switch in the fabric with Vlan1035 (private1-f1-eqiad) configured, also with an Anycast GW.
What happens is the DHCP discover, as well as being relayed to the install server by the receiving switch, is also processed as a regular broadcast within the Vlan. When it is demuxed by other switches it ultimately hits their equivalent IRB interface, triggering the same DHCP relay config. So the remote switches also end up relaying the packet, causing multiple to hit the install server. They insert different Option 82 info, however, which won't ever match a config block on the install server:
Circuit-ID SubOption 1, length 42: lsw1-e1-eqiad:'vtep.32770':private1-f1-eqiad
So despite this traffic being unwanted, it won't cause any problems. Having looked briefly it does not seem to be possible to prevent this, but we'll will have limited, if any, stretched Vlans with Anycast GWs, so it's not a big deal.
IP Filters on Routed interface
This test is to validate that IP packets are correctly filtered on direct routed links (i.e. non-IRB interfaces) in a routing-instance/VRF.
IPv4
Relevant Config
New firewall filter was added to both Spine layer devices as follows:
cmooney@lsw1-e1-eqiad> show configuration firewall family inet filter BLOCK_MS_FE1012
term BLOCK {
from {
destination-address {
208.80.154.226/32;
}
}
then {
log;
reject;
}
}
term ALLOW {
then accept;
}
cmooney@lsw1-f1-eqiad> show configuration firewall family inet filter BLOCK_MS_FE1012
term BLOCK {
from {
destination-address {
208.80.154.226/32;
}
}
then {
log;
reject;
}
}
term ALLOW {
then accept;
}
Test
The above filter was applied on each Spine as an input filter on the link from CR routers. The filter is applied both sides as traffic may arrive in via either CR router and thus we want to filter in both places to be sure to catch the packets.
cmooney@lsw1-e1-eqiad# show | compare
[edit interfaces et-0/0/48 unit 100 family inet]
+ filter {
+ input BLOCK_MS_FE1012;
+ }
cmooney@lsw1-f1-eqiad# show | compare
[edit interfaces et-0/0/48 unit 100 family inet]
+ filter {
+ input BLOCK_MS_FE1012;
+ }
We then kick off a ping from this IP, and once running we commit the above changes:
root@ms-fe1012-test:~# ping -I 208.80.154.226 1.1.1.1 PING 1.1.1.1 (1.1.1.1) from 208.80.154.226Â : 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=0.504 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=0.562 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=59 time=0.564 ms 64 bytes from 1.1.1.1: icmp_seq=4 ttl=59 time=0.530 ms 64 bytes from 1.1.1.1: icmp_seq=5 ttl=59 time=0.602 ms ^C --- 1.1.1.1 ping statistics --- 14 packets transmitted, 5 received, 64.2857% packet loss, time 13289ms rtt min/avg/max/mdev = 0.504/0.552/0.602/0.033 ms
Looking at device F1 we can see the packets were dropped as expected:
cmooney@lsw1-f1-eqiad> show firewall log | match 1.1.1.1 13:22:28 pfe R et-0/0/48.100 ICMP 1.1.1.1 208.80.154.226 13:22:27 pfe R et-0/0/48.100 ICMP 1.1.1.1 208.80.154.226 13:22:26 pfe R et-0/0/48.100 ICMP 1.1.1.1 208.80.154.226 13:22:25 pfe R et-0/0/48.100 ICMP 1.1.1.1 208.80.154.226 13:22:24 pfe R et-0/0/48.100 ICMP 1.1.1.1 208.80.154.226
IPv6
Relevant Config
Filter applied on both spine switches:
cmooney@lsw1-e1-eqiad> show configuration firewall family inet6 filter BLOCK_MS_FE1012_6
term BLOCK {
from {
destination-address {
2620:0:861:ed1a::3/128;
}
}
then {
log;
reject;
}
}
term ALLOW {
then accept;
}
cmooney@lsw1-f1-eqiad> show configuration firewall family inet6 filter BLOCK_MS_FE1012_6
term BLOCK {
from {
destination-address {
2620:0:861:ed1a::3/128;
}
}
then {
log;
reject;
}
}
term ALLOW {
then accept;
}
Tests
Similar to the v4 test the filter is applied on the Spine device links towards the Eqiad CRs (to block return internet traffic):
cmooney@lsw1-e1-eqiad# show | compare
[edit interfaces et-0/0/48 unit 100 family inet6]
+ filter {
+ input BLOCK_MS_FE1012_6;
+ }
cmooney@lsw1-f1-eqiad# show | compare
[edit interfaces et-0/0/48 unit 100 family inet6]
+ filter {
+ input BLOCK_MS_FE1012_6;
+ }
Firstly we kick off a ping, and then commit the above config on both devices:
root@ms-fe1012-test:~# ping -6 -I 2620:0:861:ed1a::3 one.one.one.one PING one.one.one.one(one.one.one.one (2606:4700:4700::1111)) from 2620:0:861:ed1a::3Â : 56 data bytes 64 bytes from one.one.one.one (2606:4700:4700::1111): icmp_seq=1 ttl=59 time=0.390 ms 64 bytes from one.one.one.one (2606:4700:4700::1111): icmp_seq=2 ttl=59 time=0.457 ms 64 bytes from one.one.one.one (2606:4700:4700::1111): icmp_seq=3 ttl=59 time=0.471 ms 64 bytes from one.one.one.one (2606:4700:4700::1111): icmp_seq=4 ttl=59 time=0.374 ms 64 bytes from one.one.one.one (2606:4700:4700::1111): icmp_seq=5 ttl=59 time=0.422 ms ^C --- one.one.one.one ping statistics --- 10 packets transmitted, 5 received, 50% packet loss, time 9154ms rtt min/avg/max/mdev = 0.374/0.422/0.471/0.037 ms
The logs on the Spine device show the packets dropped as expected:
cmooney@lsw1-f1-eqiad> show firewall log | match 2620:0:861:ed1a::3 13:38:24 pfe R et-0/0/48.100 ICMPv6 2606:4700:4700::1111 2620:0:861:ed1a::3 13:38:23 pfe R et-0/0/48.100 ICMPv6 2606:4700:4700::1111 2620:0:861:ed1a::3 13:38:22 pfe R et-0/0/48.100 ICMPv6 2606:4700:4700::1111 2620:0:861:ed1a::3 13:38:21 pfe R et-0/0/48.100 ICMPv6 2606:4700:4700::1111 2620:0:861:ed1a::3 13:38:20 pfe R et-0/0/48.100 ICMPv6 2606:4700:4700::1111 2620:0:861:ed1a::3
IP Filters on IRB interface
This test is to validate that IP packets sourced from end-hosts connected to a Vlan by access or trunk links, and sending IP packets outside their subnet via an IRB gateway on the top-of-rack, is properly filtered by ACLs/firewall filters applied to the IRB interface.
IPv4
Relevant Configuration
New firewall filter added as follows:
cmooney@lsw1-e3-eqiad> show configuration firewall family inet filter BLOCK_ELASTIC1093
term BLOCK {
from {
source-address {
10.64.132.2/32;
}
}
then {
log;
reject;
}
}
term ALLOW {
then accept;
}
Tests
The firewall was applied to irb.1033 after a ping was initiated from elastic1093-test
cmooney@lsw1-e3-eqiad# show | compare
[edit interfaces irb unit 1033 family inet]
+ filter {
+ input BLOCK_ELASTIC1093;
+ }
{master:0}[edit]
cmooney@lsw1-e3-eqiad# commit
configuration check succeeds
commit complete
When the filter was applied the ping stopped and host began to receive ICMP "packet filtered" messages (type 3, code 8) from the switch instead:
root@elastic1093-test:~# ping -I 10.64.132.2 10.3.0.1 PING 10.3.0.1 (10.3.0.1) from 10.64.132.2Â : 56(84) bytes of data. 64 bytes from 10.3.0.1: icmp_seq=1 ttl=61 time=0.234 ms 64 bytes from 10.3.0.1: icmp_seq=2 ttl=61 time=0.246 ms 64 bytes from 10.3.0.1: icmp_seq=3 ttl=61 time=0.264 ms 64 bytes from 10.3.0.1: icmp_seq=4 ttl=61 time=0.189 ms 64 bytes from 10.3.0.1: icmp_seq=5 ttl=61 time=0.257 ms From 10.64.132.1 icmp_seq=6 Packet filtered From 10.64.132.1 icmp_seq=7 Packet filtered From 10.64.132.1 icmp_seq=8 Packet filtered From 10.64.132.1 icmp_seq=9 Packet filtered
Switch firewall logs show the dropped packets too:
cmooney@lsw1-e3-eqiad> show firewall log Log : Time Filter Action Interface Protocol Src Addr Dest Addr 10:48:26 pfe R xe-0/0/21.0 TCP 10.64.132.2 10.64.132.1 10:48:22 pfe R xe-0/0/21.0 TCP 10.64.132.2 10.64.132.1 10:48:21 pfe R xe-0/0/21.0 TCP 10.64.132.2 10.64.132.1 10:48:20 pfe R xe-0/0/21.0 TCP 10.64.132.2 10.64.132.1 10:48:20 pfe R xe-0/0/21.0 TCP 10.64.132.2 10.64.132.1 10:48:19 pfe R xe-0/0/21.0 TCP 10.64.132.2 10.64.132.1 10:48:18 pfe R xe-0/0/21.0 ICMP 10.64.132.2 10.3.0.1 10:48:17 pfe R xe-0/0/21.0 ICMP 10.64.132.2 10.3.0.1 10:48:17 pfe R xe-0/0/21.0 TCP 10.64.132.2 10.64.132.1
IPv6
Same sort of test is repeated with IPv6. Test filter:
cmooney@lsw1-e3-eqiad# run show configuration firewall family inet6 filter BLOCK-ELASTIC1093_6
term BLOCK {
from {
source-address {
2620:0:861:10b:10:64:132:2/128;
}
}
then {
log;
reject;
}
}
term ALLOW {
then accept;
}
Results
Change commited as follows after ping started:
cmooney@lsw1-e3-eqiad# show | compare
[edit interfaces irb unit 1033 family inet6]
+ filter {
+ input BLOCK-ELASTIC1093_6;
+ }
{master:0}[edit interfaces irb unit 1033]
cmooney@lsw1-e3-eqiad# commit
configuration check succeeds
commit complete
Pings stopped on commit as expected, host starts getting ICMP "admin prohibited" (type 1 code 1) back from switch:
root@elastic1093-test:~# ping -I 2620:0:861:10b:10:64:132:2 authdns1001.wikimedia.org PING authdns1001.wikimedia.org(authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134)) from 2620:0:861:10b:10:64:132:2Â : 56 data bytes 64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=1 ttl=61 time=0.161 ms 64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=2 ttl=61 time=0.168 ms 64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=3 ttl=61 time=0.176 ms 64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=4 ttl=61 time=0.211 ms 64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=5 ttl=61 time=0.234 ms 64 bytes from authdns1001.wikimedia.org (2620:0:861:2:208:80:154:134): icmp_seq=6 ttl=61 time=0.180 ms From irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1) icmp_seq=7 Destination unreachable: Administratively prohibited From irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1) icmp_seq=8 Destination unreachable: Administratively prohibited From irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1) icmp_seq=9 Destination unreachable: Administratively prohibited
Firewall logs also show the drops:
cmooney@lsw1-e3-eqiad> show firewall log Log : Time Filter Action Interface Protocol Src Addr Dest Addr 11:54:14 pfe R xe-0/0/21.0 ICMPv6 2620:0:861:10b:10:64:132:2 fe80::a6e1:1a04:981:d580 11:52:44 pfe R xe-0/0/21.0 ICMPv6 2620:0:861:10b:10:64:132:2 2620:0:861:2:208:80:154:134 11:52:43 pfe R xe-0/0/21.0 ICMPv6 2620:0:861:10b:10:64:132:2 2620:0:861:2:208:80:154:134 11:52:42 pfe R xe-0/0/21.0 ICMPv6 2620:0:861:10b:10:64:132:2 2620:0:861:2:208:80:154:134 11:52:41 pfe R xe-0/0/21.0 ICMPv6 2620:0:861:10b:10:64:132:2 2620:0:861:2:208:80:154:134 11:52:40 pfe R xe-0/0/21.0 ICMPv6 2620:0:861:10b:10:64:132:2 2620:0:861:2:208:80:154:134
Failover Tests
The previous set of tests validated the configuration and operation of the network in normal circumstances. We also need to verify that failover works as expected with the given configuration.
Spine Switch Failure
The first case we need to consider it what happens if a Spine switch fails. In our topology the Spine's provide all access to the other datacenter rows, as well as the outside internet. So we need to know that if one of them fails unexpectedly traffic will continue to flow via the other.
Test Setup
A ping to an internet host was run from elastic1093-test, which is connected to Leaf switch LSW1-E3. The destination IP for the ping was one which caused LSW-E3 to ECMP it via LSW-E1, and take the path out to the internet via CR1-EQIAD. While this ping was ongoing, the power cables were removed from LSW1-E1, making the path the ping traffic was taking invalid.
Firstly we verify the traffic is routing out via E1 for the given src + dst IP:
root@elastic1093-test:~# mtr --address 208.80.154.229 -z -b -w -c 5 1.1.1.1
Start: 2022-02-18T07:59:07+0000
HOST: elastic1093-test Loss% Snt Last Avg Best Wrst StDev
1. AS??? irb-1033.lsw1-e3-eqiad.eqiad.wmnet (10.64.132.1) 0.0% 5 1.4 4.8 1.4 8.4 2.7
2. AS??? irb-1031.lsw1-e1-eqiad.eqiad.wmnet (10.64.130.1) 0.0% 5 6.1 8.8 3.3 24.7 9.0
3. AS??? et-1-0-2-100.cr1-eqiad.eqiad.wmnet (10.66.0.8) 0.0% 5 0.5 0.5 0.4 0.7 0.1
4. AS14907 ae0.cr2-eqiad.wikimedia.org (208.80.154.194) 0.0% 5 0.3 0.8 0.2 2.7 1.1
5. AS??? 13335.ash.equinix.com (206.126.237.30) 60.0% 5 1.3 8.1 1.3 14.8 9.6
6. AS13335 172.70.172.2 0.0% 5 7.5 3.0 1.6 7.5 2.6
7. AS13335 one.one.one.one (1.1.1.1) 0.0% 5 0.7 0.7 0.7 0.7 0.0
Then we kick off our ping, asking on-site staff to power down E1 once it's running:
root@elastic1093-test:~# ping 1.1.1.1 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=0.574 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=0.605 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=59 time=0.625 ms 64 bytes from 1.1.1.1: icmp_seq=4 ttl=59 time=0.629 ms 64 bytes from 1.1.1.1: icmp_seq=5 ttl=59 time=0.610 ms 64 bytes from 1.1.1.1: icmp_seq=6 ttl=59 time=0.630 ms 64 bytes from 1.1.1.1: icmp_seq=7 ttl=59 time=0.523 ms 64 bytes from 1.1.1.1: icmp_seq=8 ttl=59 time=0.564 ms 64 bytes from 1.1.1.1: icmp_seq=9 ttl=59 time=0.642 ms 64 bytes from 1.1.1.1: icmp_seq=10 ttl=59 time=0.635 ms 64 bytes from 1.1.1.1: icmp_seq=11 ttl=59 time=0.581 ms 64 bytes from 1.1.1.1: icmp_seq=12 ttl=59 time=0.544 ms 64 bytes from 1.1.1.1: icmp_seq=13 ttl=59 time=0.634 ms 64 bytes from 1.1.1.1: icmp_seq=14 ttl=59 time=0.581 ms 64 bytes from 1.1.1.1: icmp_seq=15 ttl=59 time=0.586 ms 64 bytes from 1.1.1.1: icmp_seq=16 ttl=59 time=0.558 ms 64 bytes from 1.1.1.1: icmp_seq=17 ttl=59 time=0.609 ms 64 bytes from 1.1.1.1: icmp_seq=18 ttl=59 time=0.557 ms 64 bytes from 1.1.1.1: icmp_seq=19 ttl=59 time=0.624 ms 64 bytes from 1.1.1.1: icmp_seq=20 ttl=59 time=0.565 ms 64 bytes from 1.1.1.1: icmp_seq=21 ttl=59 time=0.536 ms 64 bytes from 1.1.1.1: icmp_seq=22 ttl=59 time=0.596 ms 64 bytes from 1.1.1.1: icmp_seq=23 ttl=59 time=0.589 ms 64 bytes from 1.1.1.1: icmp_seq=24 ttl=59 time=0.609 ms 64 bytes from 1.1.1.1: icmp_seq=25 ttl=59 time=0.525 ms 64 bytes from 1.1.1.1: icmp_seq=26 ttl=59 time=0.626 ms 64 bytes from 1.1.1.1: icmp_seq=27 ttl=59 time=0.584 ms 64 bytes from 1.1.1.1: icmp_seq=28 ttl=59 time=0.541 ms 64 bytes from 1.1.1.1: icmp_seq=29 ttl=59 time=0.722 ms 64 bytes from 1.1.1.1: icmp_seq=30 ttl=59 time=0.628 ms 64 bytes from 1.1.1.1: icmp_seq=31 ttl=59 time=0.572 ms 64 bytes from 1.1.1.1: icmp_seq=32 ttl=59 time=0.512 ms 64 bytes from 1.1.1.1: icmp_seq=33 ttl=59 time=0.633 ms 64 bytes from 1.1.1.1: icmp_seq=34 ttl=59 time=0.611 ms 64 bytes from 1.1.1.1: icmp_seq=35 ttl=59 time=0.618 ms 64 bytes from 1.1.1.1: icmp_seq=36 ttl=59 time=0.579 ms 64 bytes from 1.1.1.1: icmp_seq=37 ttl=59 time=0.716 ms 64 bytes from 1.1.1.1: icmp_seq=38 ttl=59 time=0.579 ms 64 bytes from 1.1.1.1: icmp_seq=39 ttl=59 time=0.561 ms 64 bytes from 1.1.1.1: icmp_seq=40 ttl=59 time=0.537 ms 64 bytes from 1.1.1.1: icmp_seq=41 ttl=59 time=0.645 ms 64 bytes from 1.1.1.1: icmp_seq=42 ttl=59 time=0.521 ms 64 bytes from 1.1.1.1: icmp_seq=43 ttl=59 time=0.559 ms 64 bytes from 1.1.1.1: icmp_seq=44 ttl=59 time=0.602 ms 64 bytes from 1.1.1.1: icmp_seq=45 ttl=59 time=0.526 ms 64 bytes from 1.1.1.1: icmp_seq=46 ttl=59 time=0.578 ms 64 bytes from 1.1.1.1: icmp_seq=47 ttl=59 time=0.977 ms 64 bytes from 1.1.1.1: icmp_seq=48 ttl=59 time=0.481 ms 64 bytes from 1.1.1.1: icmp_seq=49 ttl=59 time=0.602 ms 64 bytes from 1.1.1.1: icmp_seq=50 ttl=59 time=0.553 ms 64 bytes from 1.1.1.1: icmp_seq=51 ttl=59 time=0.708 ms 64 bytes from 1.1.1.1: icmp_seq=52 ttl=59 time=0.494 ms 64 bytes from 1.1.1.1: icmp_seq=53 ttl=59 time=0.556 ms 64 bytes from 1.1.1.1: icmp_seq=54 ttl=59 time=0.513 ms 64 bytes from 1.1.1.1: icmp_seq=55 ttl=59 time=0.544 ms 64 bytes from 1.1.1.1: icmp_seq=56 ttl=59 time=0.487 ms 64 bytes from 1.1.1.1: icmp_seq=57 ttl=59 time=0.481 ms 64 bytes from 1.1.1.1: icmp_seq=58 ttl=59 time=0.467 ms 64 bytes from 1.1.1.1: icmp_seq=59 ttl=59 time=0.497 ms 64 bytes from 1.1.1.1: icmp_seq=60 ttl=59 time=0.490 ms 64 bytes from 1.1.1.1: icmp_seq=61 ttl=59 time=0.564 ms 64 bytes from 1.1.1.1: icmp_seq=62 ttl=59 time=0.505 ms 64 bytes from 1.1.1.1: icmp_seq=63 ttl=59 time=0.529 ms 64 bytes from 1.1.1.1: icmp_seq=64 ttl=59 time=0.527 ms 64 bytes from 1.1.1.1: icmp_seq=65 ttl=59 time=0.556 ms 64 bytes from 1.1.1.1: icmp_seq=66 ttl=59 time=0.592 ms 64 bytes from 1.1.1.1: icmp_seq=67 ttl=59 time=0.541 ms 64 bytes from 1.1.1.1: icmp_seq=68 ttl=59 time=0.561 ms 64 bytes from 1.1.1.1: icmp_seq=69 ttl=59 time=0.475 ms 64 bytes from 1.1.1.1: icmp_seq=70 ttl=59 time=0.586 ms 64 bytes from 1.1.1.1: icmp_seq=71 ttl=59 time=0.539 ms 64 bytes from 1.1.1.1: icmp_seq=72 ttl=59 time=0.525 ms 64 bytes from 1.1.1.1: icmp_seq=73 ttl=59 time=0.635 ms 64 bytes from 1.1.1.1: icmp_seq=74 ttl=59 time=0.541 ms 64 bytes from 1.1.1.1: icmp_seq=75 ttl=59 time=0.538 ms 64 bytes from 1.1.1.1: icmp_seq=76 ttl=59 time=0.568 ms 64 bytes from 1.1.1.1: icmp_seq=77 ttl=59 time=0.553 ms ^C --- 1.1.1.1 ping statistics --- 77 packets transmitted, 77 received, 0% packet loss, time 77791ms rtt min/avg/max/mdev = 0.467/0.576/0.977/0.070 ms
Stopping the ping a short time after the confirmation LSW1-E1 was powered off we see that we have 0 pings lost, a good sign. We can also see in a traceroute that the traffic is now flowing out via LSW1-F1 to CR2-EQIAD:
root@elastic1093-test:~# mtr --address 208.80.154.229 -z -b -w -c 5 1.1.1.1
Start: 2022-02-18T08:08:11+0000
HOST: elastic1093-test Loss% Snt Last Avg Best Wrst StDev
1. AS??? irb-1033.lsw1-e3-eqiad.eqiad.wmnet (10.64.132.1) 0.0% 5 7.4 3.8 1.4 7.4 2.4
2. AS??? irb-1035.lsw1-f1-eqiad.eqiad.wmnet (10.64.134.1) 0.0% 5 8.3 5.3 3.2 8.3 2.0
3. AS??? et-1-0-2-100.cr2-eqiad.eqiad.wmnet (10.66.0.10) 0.0% 5 0.6 0.8 0.4 1.5 0.5
4. AS??? 13335.ash.equinix.com (206.126.237.30) 40.0% 5 1.6 7.7 1.6 19.5 10.2
5. AS13335 172.70.172.2 0.0% 5 2.7 11.2 1.2 45.3 19.1
6. AS13335 one.one.one.one (1.1.1.1) 0.0% 5 0.7 1.0 0.7 2.4 0.8
So all looks good. The other thing we want to do is verify things return to normal after power is restored, so we kick off another ping and ask on-site to restore power to LSW1-E1:
root@elastic1093-test:~# ping 1.1.1.1 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=0.504 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=0.660 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=59 time=0.524 ms 64 bytes from 1.1.1.1: icmp_seq=4 ttl=59 time=0.607 ms 64 bytes from 1.1.1.1: icmp_seq=5 ttl=59 time=0.553 ms 64 bytes from 1.1.1.1: icmp_seq=6 ttl=59 time=0.543 ms 64 bytes from 1.1.1.1: icmp_seq=7 ttl=59 time=0.566 ms 64 bytes from 1.1.1.1: icmp_seq=8 ttl=59 time=0.565 ms 64 bytes from 1.1.1.1: icmp_seq=9 ttl=59 time=0.559 ms 64 bytes from 1.1.1.1: icmp_seq=10 ttl=59 time=0.593 ms 64 bytes from 1.1.1.1: icmp_seq=11 ttl=59 time=0.654 ms 64 bytes from 1.1.1.1: icmp_seq=12 ttl=59 time=0.565 ms 64 bytes from 1.1.1.1: icmp_seq=13 ttl=59 time=0.671 ms 64 bytes from 1.1.1.1: icmp_seq=14 ttl=59 time=0.576 ms 64 bytes from 1.1.1.1: icmp_seq=15 ttl=59 time=0.605 ms 64 bytes from 1.1.1.1: icmp_seq=16 ttl=59 time=0.565 ms 64 bytes from 1.1.1.1: icmp_seq=17 ttl=59 time=0.575 ms 64 bytes from 1.1.1.1: icmp_seq=18 ttl=59 time=0.700 ms 64 bytes from 1.1.1.1: icmp_seq=19 ttl=59 time=0.573 ms 64 bytes from 1.1.1.1: icmp_seq=20 ttl=59 time=0.632 ms 64 bytes from 1.1.1.1: icmp_seq=21 ttl=59 time=0.580 ms 64 bytes from 1.1.1.1: icmp_seq=22 ttl=59 time=0.609 ms 64 bytes from 1.1.1.1: icmp_seq=23 ttl=59 time=0.649 ms 64 bytes from 1.1.1.1: icmp_seq=24 ttl=59 time=0.639 ms 64 bytes from 1.1.1.1: icmp_seq=25 ttl=59 time=0.579 ms 64 bytes from 1.1.1.1: icmp_seq=26 ttl=59 time=0.618 ms 64 bytes from 1.1.1.1: icmp_seq=27 ttl=59 time=0.597 ms 64 bytes from 1.1.1.1: icmp_seq=28 ttl=59 time=0.639 ms 64 bytes from 1.1.1.1: icmp_seq=29 ttl=59 time=0.706 ms 64 bytes from 1.1.1.1: icmp_seq=30 ttl=59 time=0.557 ms 64 bytes from 1.1.1.1: icmp_seq=31 ttl=59 time=0.547 ms 64 bytes from 1.1.1.1: icmp_seq=32 ttl=59 time=0.595 ms 64 bytes from 1.1.1.1: icmp_seq=33 ttl=59 time=0.583 ms 64 bytes from 1.1.1.1: icmp_seq=34 ttl=59 time=0.611 ms 64 bytes from 1.1.1.1: icmp_seq=35 ttl=59 time=0.596 ms 64 bytes from 1.1.1.1: icmp_seq=36 ttl=59 time=0.591 ms 64 bytes from 1.1.1.1: icmp_seq=37 ttl=59 time=0.624 ms 64 bytes from 1.1.1.1: icmp_seq=38 ttl=59 time=0.554 ms 64 bytes from 1.1.1.1: icmp_seq=39 ttl=59 time=0.606 ms 64 bytes from 1.1.1.1: icmp_seq=40 ttl=59 time=0.588 ms 64 bytes from 1.1.1.1: icmp_seq=41 ttl=59 time=0.620 ms 64 bytes from 1.1.1.1: icmp_seq=42 ttl=59 time=0.562 ms 64 bytes from 1.1.1.1: icmp_seq=43 ttl=59 time=0.602 ms 64 bytes from 1.1.1.1: icmp_seq=44 ttl=59 time=0.592 ms 64 bytes from 1.1.1.1: icmp_seq=45 ttl=59 time=0.615 ms 64 bytes from 1.1.1.1: icmp_seq=46 ttl=59 time=0.606 ms 64 bytes from 1.1.1.1: icmp_seq=47 ttl=59 time=0.550 ms 64 bytes from 1.1.1.1: icmp_seq=48 ttl=59 time=0.593 ms 64 bytes from 1.1.1.1: icmp_seq=49 ttl=59 time=0.621 ms 64 bytes from 1.1.1.1: icmp_seq=50 ttl=59 time=0.560 ms 64 bytes from 1.1.1.1: icmp_seq=51 ttl=59 time=0.597 ms 64 bytes from 1.1.1.1: icmp_seq=52 ttl=59 time=0.626 ms 64 bytes from 1.1.1.1: icmp_seq=53 ttl=59 time=0.577 ms 64 bytes from 1.1.1.1: icmp_seq=54 ttl=59 time=0.558 ms 64 bytes from 1.1.1.1: icmp_seq=55 ttl=59 time=0.588 ms 64 bytes from 1.1.1.1: icmp_seq=56 ttl=59 time=0.544 ms 64 bytes from 1.1.1.1: icmp_seq=57 ttl=59 time=0.564 ms 64 bytes from 1.1.1.1: icmp_seq=58 ttl=59 time=0.643 ms 64 bytes from 1.1.1.1: icmp_seq=59 ttl=59 time=0.569 ms 64 bytes from 1.1.1.1: icmp_seq=60 ttl=59 time=0.573 ms 64 bytes from 1.1.1.1: icmp_seq=61 ttl=59 time=0.572 ms 64 bytes from 1.1.1.1: icmp_seq=62 ttl=59 time=0.599 ms 64 bytes from 1.1.1.1: icmp_seq=63 ttl=59 time=0.596 ms 64 bytes from 1.1.1.1: icmp_seq=64 ttl=59 time=0.580 ms 64 bytes from 1.1.1.1: icmp_seq=65 ttl=59 time=0.643 ms 64 bytes from 1.1.1.1: icmp_seq=66 ttl=59 time=0.601 ms 64 bytes from 1.1.1.1: icmp_seq=67 ttl=59 time=0.549 ms 64 bytes from 1.1.1.1: icmp_seq=68 ttl=59 time=0.701 ms 64 bytes from 1.1.1.1: icmp_seq=69 ttl=59 time=0.653 ms 64 bytes from 1.1.1.1: icmp_seq=70 ttl=59 time=0.631 ms 64 bytes from 1.1.1.1: icmp_seq=71 ttl=59 time=0.632 ms 64 bytes from 1.1.1.1: icmp_seq=72 ttl=59 time=0.617 ms 64 bytes from 1.1.1.1: icmp_seq=73 ttl=59 time=0.601 ms 64 bytes from 1.1.1.1: icmp_seq=74 ttl=59 time=0.597 ms 64 bytes from 1.1.1.1: icmp_seq=75 ttl=59 time=0.668 ms 64 bytes from 1.1.1.1: icmp_seq=76 ttl=59 time=0.572 ms 64 bytes from 1.1.1.1: icmp_seq=77 ttl=59 time=0.601 ms 64 bytes from 1.1.1.1: icmp_seq=78 ttl=59 time=0.637 ms 64 bytes from 1.1.1.1: icmp_seq=79 ttl=59 time=0.589 ms 64 bytes from 1.1.1.1: icmp_seq=80 ttl=59 time=0.571 ms 64 bytes from 1.1.1.1: icmp_seq=81 ttl=59 time=0.608 ms 64 bytes from 1.1.1.1: icmp_seq=82 ttl=59 time=0.674 ms 64 bytes from 1.1.1.1: icmp_seq=83 ttl=59 time=0.583 ms 64 bytes from 1.1.1.1: icmp_seq=84 ttl=59 time=0.629 ms 64 bytes from 1.1.1.1: icmp_seq=85 ttl=59 time=0.629 ms 64 bytes from 1.1.1.1: icmp_seq=86 ttl=59 time=0.592 ms 64 bytes from 1.1.1.1: icmp_seq=87 ttl=59 time=0.610 ms 64 bytes from 1.1.1.1: icmp_seq=88 ttl=59 time=0.542 ms ^C --- 1.1.1.1 ping statistics --- 88 packets transmitted, 88 received, 0% packet loss, time 89071ms rtt min/avg/max/mdev = 0.504/0.598/0.706/0.039 ms
Again no packets lost, and when we do a trace out we can see the traffic is flowing via lsw1-e1 again:
root@elastic1093-test:~# mtr --address 208.80.154.229 -z -b -w -c 5 1.1.1.1
Start: 2022-02-18T08:15:14+0000
HOST: elastic1093-test Loss% Snt Last Avg Best Wrst StDev
1. AS??? irb-1033.lsw1-e3-eqiad.eqiad.wmnet (10.64.132.1) 0.0% 5 1.6 5.1 1.6 8.1 3.2
2. AS??? irb-1031.lsw1-e1-eqiad.eqiad.wmnet (10.64.130.1) 0.0% 5 6.1 13.9 0.8 51.9 21.6
3. AS??? et-1-0-2-100.cr1-eqiad.eqiad.wmnet (10.66.0.8) 0.0% 5 0.5 0.5 0.4 0.6 0.1
4. AS14907 ae0.cr2-eqiad.wikimedia.org (208.80.154.194) 0.0% 5 0.3 0.6 0.3 1.6 0.6
5. AS??? Â ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
6. AS13335 172.70.172.2 0.0% 5 2.2 1.8 1.2 2.2 0.4
7. AS13335 one.one.one.one (1.1.1.1) 0.0% 5 0.7 0.7 0.7 0.7 0.0
Leaf to Spine Link Failure
This was not tested separately as the 'Spine Link Failure' effectively, from the Leaf switch's point of view, has the same effect as the link to the Spine going down. It has no way to tell if it's a device failure or just the link, so it will behave in the same way. As such there is no need to run a separate test.
Spine to CR Link Failure
In this test we want to pull the cable running from lsw1-e1 to cr1-eqiad, and observe what happens. Testing is again run from elastic1093-test, connected to lsw1-e3. LSW1-E3 will not notice any physical change or difference in IGP topology, however it should get an EVPN BGP WITHDRAW for the default route it was receiving from LSW1-E1, with it's own VTEP IP as next-hop. LSW1-E3 will still receive a default BGP route from LSW1-E1, however this will be the route it knows from LSW1-F1, and the next-hop VTEP will be that of the other switch.
This process should happen very quickly, and all traffic flow from LSW1-E3 to LSW1-F1 shortly after the cable pull and resulting BGP WITHDRAW.
Results
Firstly we establish that our ping routes out from LSW1-E3 via LSW1-E1 for our chosen src and destination:
root@elastic1093-test:~# mtr --address 2620:0:861:ed1a::6 -z -b -w -c 5 www.ietf.org
Start: 2022-02-18T08:15:56+0000
HOST: elastic1093-test Loss% Snt Last Avg Best Wrst StDev
1. AS14907 irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1) 0.0% 5 0.5 0.5 0.5 0.6 0.0
2. AS14907 irb-1039.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:100::1) 0.0% 5 0.5 0.5 0.5 0.6 0.0
3. AS14907 et-1-0-2-100.cr1-eqiad.eqiad.wmnet (2620:0:861:fe07::1) 0.0% 5 0.8 0.6 0.4 0.8 0.2
4. AS14907 ae0.cr2-eqiad.wikimedia.org (2620:0:861:fe00::2) 0.0% 5 2.1 0.9 0.4 2.1 0.7
5. AS??? 13335.ash.equinix.com (2001:504:0:2:0:1:3335:1) 80.0% 5 2.1 2.1 2.1 2.1 0.0
6. AS13335 2400:cb00:354:3:: 0.0% 5 0.4 3.9 0.4 8.2 3.5
7. AS13335 2606:4700::6810:2d63 0.0% 5 0.5 1.3 0.4 4.6 1.8
Next we kick off the ping, and ask DC-Ops to pull the cable in the meantime:
root@elastic1093-test:~# ping6 -I 2620:0:861:ed1a::6 www.ietf.org PING www.ietf.org(2606:4700::6810:2d63 (2606:4700::6810:2d63)) from 2620:0:861:ed1a::6Â : 56 data bytes 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=1 ttl=59 time=4.69 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=2 ttl=59 time=0.440 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=3 ttl=59 time=0.364 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=4 ttl=59 time=0.434 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=5 ttl=59 time=0.374 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=6 ttl=59 time=0.470 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=7 ttl=59 time=0.454 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=8 ttl=59 time=0.489 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=9 ttl=59 time=0.376 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=10 ttl=59 time=0.709 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=11 ttl=59 time=4.29 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=12 ttl=59 time=1.28 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=13 ttl=59 time=0.458 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=14 ttl=59 time=0.414 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=15 ttl=59 time=14.4 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=16 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=17 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=18 ttl=59 time=0.463 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=19 ttl=59 time=0.446 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=20 ttl=59 time=0.457 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=21 ttl=59 time=0.446 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=22 ttl=59 time=0.432 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=23 ttl=59 time=0.458 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=24 ttl=59 time=0.440 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=25 ttl=59 time=0.426 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=26 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=27 ttl=59 time=0.434 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=28 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=29 ttl=59 time=0.340 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=30 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=31 ttl=59 time=0.461 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=32 ttl=59 time=0.432 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=33 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=34 ttl=59 time=0.427 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=35 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=36 ttl=59 time=0.425 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=37 ttl=59 time=0.346 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=38 ttl=59 time=0.435 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=39 ttl=59 time=0.425 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=40 ttl=59 time=0.439 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=41 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=42 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=43 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=44 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=45 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=46 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=47 ttl=59 time=0.425 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=48 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=49 ttl=59 time=0.427 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=50 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=51 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=52 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=53 ttl=59 time=0.446 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=54 ttl=59 time=0.351 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=55 ttl=59 time=0.437 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=56 ttl=59 time=0.429 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=57 ttl=59 time=0.452 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=58 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=59 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=60 ttl=59 time=0.457 ms ^C --- www.ietf.org ping statistics --- 60 packets transmitted, 60 received, 0% packet loss, time 59427ms rtt min/avg/max/mdev = 0.340/0.814/14.417/1.917 ms
Zero packet's lost again which is good, let's verify the traffic is taking the new path:
root@elastic1093-test:~# mtr --address 2620:0:861:ed1a::6 -z -b -w -c 5 www.ietf.org
Start: 2022-02-18T08:21:40+0000
HOST: elastic1093-test Loss% Snt Last Avg Best Wrst StDev
1. AS14907 irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1) 0.0% 5 0.5 0.5 0.5 0.6 0.0
2. AS14907 irb-1035.lsw1-f1-eqiad.eqiad.wmnet (2620:0:861:10d::1) 0.0% 5 0.7 10.2 0.6 48.3 21.3
3. AS14907 et-1-0-2-100.cr2-eqiad.eqiad.wmnet (2620:0:861:fe08::1) 0.0% 5 0.7 0.5 0.4 0.7 0.1
4. AS??? 13335.ash.equinix.com (2001:504:0:2:0:1:3335:1) 40.0% 5 12.0 7.5 1.5 12.0 5.4
5. AS13335 2400:cb00:350:3:: 0.0% 5 4.7 2.0 1.1 4.7 1.5
6. AS13335 2606:4700::6810:2c63 0.0% 5 0.5 0.4 0.4 0.5 0.0
All looking good so we ask on-site to restore the link, and monitor to make sure there are no issues:
root@elastic1093-test:~# ping6 -I 2620:0:861:ed1a::6 www.ietf.org PING www.ietf.org(2606:4700::6810:2d63 (2606:4700::6810:2d63)) from 2620:0:861:ed1a::6Â : 56 data bytes 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=1 ttl=59 time=0.370 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=2 ttl=59 time=0.353 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=3 ttl=59 time=0.434 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=4 ttl=59 time=0.442 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=5 ttl=59 time=0.428 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=6 ttl=59 time=0.438 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=7 ttl=59 time=0.341 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=8 ttl=59 time=0.335 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=9 ttl=59 time=0.361 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=10 ttl=59 time=0.425 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=11 ttl=59 time=5.31 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=12 ttl=59 time=0.353 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=13 ttl=59 time=0.448 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=14 ttl=59 time=0.420 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=15 ttl=59 time=0.447 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=16 ttl=59 time=0.425 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=17 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=18 ttl=59 time=0.335 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=19 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=20 ttl=59 time=0.427 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=21 ttl=59 time=0.606 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=22 ttl=59 time=0.475 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=23 ttl=59 time=0.444 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=24 ttl=59 time=0.425 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=25 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=26 ttl=59 time=0.444 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=27 ttl=59 time=0.338 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=28 ttl=59 time=0.342 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=29 ttl=59 time=0.426 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=30 ttl=59 time=0.331 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=31 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=32 ttl=59 time=0.425 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=33 ttl=59 time=0.463 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=34 ttl=59 time=0.496 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=35 ttl=59 time=0.538 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=36 ttl=59 time=0.420 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=37 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=38 ttl=59 time=0.794 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=39 ttl=59 time=0.608 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=40 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=41 ttl=59 time=0.438 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=42 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=43 ttl=59 time=0.545 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=44 ttl=59 time=0.542 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=45 ttl=59 time=0.426 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=46 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=47 ttl=59 time=0.483 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=48 ttl=59 time=0.421 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=49 ttl=59 time=0.446 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=50 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=51 ttl=59 time=0.434 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=52 ttl=59 time=0.440 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=53 ttl=59 time=0.448 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=54 ttl=59 time=0.638 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=55 ttl=59 time=0.452 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=56 ttl=59 time=0.475 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=57 ttl=59 time=0.466 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=58 ttl=59 time=0.401 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=59 ttl=59 time=0.522 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=60 ttl=59 time=1.22 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=61 ttl=59 time=0.441 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=62 ttl=59 time=0.448 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=63 ttl=59 time=0.426 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=64 ttl=59 time=0.340 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=65 ttl=59 time=0.441 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=66 ttl=59 time=0.547 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=67 ttl=59 time=0.362 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=68 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=69 ttl=59 time=0.373 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=70 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=71 ttl=59 time=0.441 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=72 ttl=59 time=0.520 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=73 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=74 ttl=59 time=0.641 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=75 ttl=59 time=0.432 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=76 ttl=59 time=1.78 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=77 ttl=59 time=0.440 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=78 ttl=59 time=0.428 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=79 ttl=59 time=0.421 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=80 ttl=59 time=0.430 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=81 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=82 ttl=59 time=0.380 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=83 ttl=59 time=0.451 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=84 ttl=59 time=0.440 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=85 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=86 ttl=59 time=0.387 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=87 ttl=59 time=2.15 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=88 ttl=59 time=0.473 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=89 ttl=59 time=0.523 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=90 ttl=59 time=0.444 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=91 ttl=59 time=0.421 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=92 ttl=59 time=0.452 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=93 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=94 ttl=59 time=0.457 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=95 ttl=59 time=0.351 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=96 ttl=59 time=0.518 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=97 ttl=59 time=0.551 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=98 ttl=59 time=0.439 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=99 ttl=59 time=0.440 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=100 ttl=59 time=0.446 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=101 ttl=59 time=0.445 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=102 ttl=59 time=0.501 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=103 ttl=59 time=0.455 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=104 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=105 ttl=59 time=0.443 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=106 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=107 ttl=59 time=0.444 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=108 ttl=59 time=0.457 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=109 ttl=59 time=0.443 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=110 ttl=59 time=0.451 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=111 ttl=59 time=0.442 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=112 ttl=59 time=0.335 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=113 ttl=59 time=0.421 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=114 ttl=59 time=0.452 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=115 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=116 ttl=59 time=0.718 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=117 ttl=59 time=0.377 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=118 ttl=59 time=0.416 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=119 ttl=59 time=0.444 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=120 ttl=59 time=3.95 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=121 ttl=59 time=0.529 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=122 ttl=59 time=0.447 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=123 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=124 ttl=59 time=0.444 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=125 ttl=59 time=0.435 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=126 ttl=59 time=0.466 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=127 ttl=59 time=0.451 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=128 ttl=59 time=0.440 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=129 ttl=59 time=0.435 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=130 ttl=59 time=0.457 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=131 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=132 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=133 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=134 ttl=59 time=0.460 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=135 ttl=59 time=0.336 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=136 ttl=59 time=0.460 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=137 ttl=59 time=0.449 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=138 ttl=59 time=0.420 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=139 ttl=59 time=0.421 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=140 ttl=59 time=0.465 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=141 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=142 ttl=59 time=0.435 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=143 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=144 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=145 ttl=59 time=0.462 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=146 ttl=59 time=0.443 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=147 ttl=59 time=0.449 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=148 ttl=59 time=0.442 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=149 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=150 ttl=59 time=0.353 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=151 ttl=59 time=0.463 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=152 ttl=59 time=0.428 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=153 ttl=59 time=0.466 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=154 ttl=59 time=0.441 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=155 ttl=59 time=0.455 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=156 ttl=59 time=0.453 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=157 ttl=59 time=0.422 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=158 ttl=59 time=0.461 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=159 ttl=59 time=0.387 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=160 ttl=59 time=0.434 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=161 ttl=59 time=0.449 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=162 ttl=59 time=0.441 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=163 ttl=59 time=0.364 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=164 ttl=59 time=0.431 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=165 ttl=59 time=0.428 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=166 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=167 ttl=59 time=0.433 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=168 ttl=59 time=0.465 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=169 ttl=59 time=0.439 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=170 ttl=59 time=0.456 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=171 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=172 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=173 ttl=59 time=0.449 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=174 ttl=59 time=0.499 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=175 ttl=59 time=0.449 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=176 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=177 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=178 ttl=59 time=4.20 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=179 ttl=59 time=0.495 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=180 ttl=59 time=0.357 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=181 ttl=59 time=0.433 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=182 ttl=59 time=0.441 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=183 ttl=59 time=0.446 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=184 ttl=59 time=0.354 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=185 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=186 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=187 ttl=59 time=0.434 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=188 ttl=59 time=0.424 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=189 ttl=59 time=0.425 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=190 ttl=59 time=0.481 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=191 ttl=59 time=0.459 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=192 ttl=59 time=0.445 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=193 ttl=59 time=0.423 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=194 ttl=59 time=0.535 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=195 ttl=59 time=0.440 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=196 ttl=59 time=0.456 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=197 ttl=59 time=0.363 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=198 ttl=59 time=0.421 ms 64 bytes from 2606:4700::6810:2d63 (2606:4700::6810:2d63): icmp_seq=199 ttl=59 time=0.426 ms ^C --- www.ietf.org ping statistics --- 199 packets transmitted, 199 received, 0% packet loss, time 199094ms rtt min/avg/max/mdev = 0.331/0.521/5.307/0.525 ms
root@elastic1093-test:~# mtr --address 2620:0:861:ed1a::6 -z -b -w -c 5 www.ietf.org
Start: 2022-02-18T08:25:48+0000
HOST: elastic1093-test Loss% Snt Last Avg Best Wrst StDev
1. AS14907 irb-1033.lsw1-e3-eqiad.eqiad.wmnet (2620:0:861:10b::1) 0.0% 5 0.6 1.7 0.5 6.2 2.5
2. AS14907 irb-1039.lsw1-e1-eqiad.eqiad.wmnet (2620:0:861:100::1) 0.0% 5 0.7 1.0 0.5 2.8 1.0
3. AS14907 et-1-0-2-100.cr1-eqiad.eqiad.wmnet (2620:0:861:fe07::1) 0.0% 5 0.4 0.5 0.4 0.6 0.1
4. AS14907 ae0.cr2-eqiad.wikimedia.org (2620:0:861:fe00::2) 0.0% 5 0.5 0.6 0.3 1.2 0.4
5. AS??? 13335.ash.equinix.com (2001:504:0:2:0:1:3335:1) 80.0% 5 29.6 29.6 29.6 29.6 0.0
6. AS13335 2400:cb00:354:3:: 0.0% 5 1.1 1.8 0.5 5.0 1.8
7. AS13335 2606:4700::6810:2d63 0.0% 5 0.5 0.6 0.5 0.8 0.1
Again no lost packets, and we can see that by the time we are finished traffic is back on the original path.