SONiC/Leaf-spine testing
< SONiC
(moved from SONiC)
Leaf/spine configuration
We are going to use the diagram below to setup the leaf/spine. Each lead switch has 1 connection to each spine switch. There is no connection between the spines. The first setup is to identify the interfaces on each switch.
Interface mapping
show interfaces description
on S5232F-ON (Spine)
Interface Oper Admin Alias Description ----------- ------ ------- ------- ------------- Ethernet0 down down Eth1/1 N/A Ethernet4 down down Eth1/2 N/A Ethernet8 down down Eth1/3 N/A Ethernet12 down down Eth1/4 N/A Ethernet16 down down Eth1/5 N/A Ethernet20 down down Eth1/6 N/A - - Ethernet120 down down Eth1/31 N/A Ethernet124 down down Eth1/32 N/A Ethernet128 down down Eth1/33 N/A Ethernet129 down down Eth1/34 N/A
on both spines switches, we will use interfaces Ethernet120 on Sonic side which is interface E1/31 on Dell switch and Ethernet124 which is E1/32
On S5248F-ON (leaf)
leaf1# show interface description
Interface Oper Admin Alias Description ----------- ------ ------- ------- ------------- Ethernet0 down down Eth1/1 N/A Ethernet1 down down Eth1/2 N/A Ethernet2 down down Eth1/3 N/A Ethernet3 down down Eth1/4 N/A - - Ethernet72 down down Eth1/55 N/A Ethernet76 down down Eth1/56 N/A
On the leaf's we will use interfaces Ethernet72 and Ethernet76 which are E1/55 and E1/56 on the Dell switch
Now that we have the interface mapping from Sonic to Dell it is time to setup IP address on each of those interfaces. Below is the diagram showing the IP's and network that we will use for the leaf/spine setup.
Interface configuration
spine1
sonic-cli configure terminal interface Ethernet120 description dell-leaf2 no shutdown ip address 172.16.1.9/30 interface Ethernet124 description dell-leaf1 no shutdown ip address 172.16.1.1/30 end write memory
spine2
sonic-cli configure terminal interface Ethernet120 description dell-leaf2 no shutdown ip address 172.16.1.14/30 interface Ethernet124 description dell-leaf1 no shutdown ip address 172.16.1.6/30 end write memory
leaf1
sonic-cli configure terminal interface Ethernet72 description dell-spine2 no shutdown ip address 172.16.1.6/30 interface Ethernet76 description dell-spine1 no shutdown ip address 172.16.1.2/30
leaf2 sonic-cli configure terminal interface Ethernet72 description dell-spine2 no shutdown ip address 172.16.1.14/30 interface Ethernet76 description dell-spine1 no shutdown ip address 172.16.1.10/30
Check connectivity between spine1 and leaf1
dell-spine1# ping 172.16.1.2 PING 172.16.1.2 (172.16.1.2) 56(84) bytes of data. 64 bytes from 172.16.1.2: icmp_seq=1 ttl=64 time=0.348 ms 64 bytes from 172.16.1.2: icmp_seq=2 ttl=64 time=0.274 ms 64 bytes from 172.16.1.2: icmp_seq=3 ttl=64 time=0.268 ms
dell-leaf1# ping 172.16.1.1 PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data. 64 bytes from 172.16.1.1: icmp_seq=1 ttl=64 time=0.339 ms 64 bytes from 172.16.1.1: icmp_seq=2 ttl=64 time=0.303 ms
dell-spine2# ping 172.16.1.14 PING 172.16.1.14 (172.16.1.14) 56(84) bytes of data. 64 bytes from 172.16.1.14: icmp_seq=1 ttl=64 time=0.356 ms 64 bytes from 172.16.1.14: icmp_seq=2 ttl=64 time=0.296 ms
Now that we have connectivity, the next step is to setup BGP
BGP configuration
spine1
sonic-cli configure terminal router bgp 65030 router-id 10.0.1.13 address-family ipv4 unicast redistribute connected neighbor 172.16.1.10 remote-as 65033 address-family ipv4 unicast activate neighbor 172.16.1.2 remote-as 65032 address-family ipv4 unicast activate end write memory
spine2
sonic-cli configure terminal router bgp 65030 router-id 10.0.1.14 address-family ipv4 unicast redistribute connected neighbor 172.16.1.14 remote-as 65033 address-family ipv4 unicast activate neighbor 172.16.1.6 remote-as 65032 address-family ipv4 unicast activate end write memory
leaf 1
router bgp 65032 router-id 10.0.1.24 address-family ipv4 unicast redistribute connected neighbor 172.16.1.1 remote-as 65030 address-family ipv4 unicast activate neighbor 172.16.1.5 remote-as 65030 address-family ipv4 unicast activate end write memory
leaf 2
router bgp 65033 router-id 10.0.1.25 address-family ipv4 unicast redistribute connected neighbor 172.16.1.9 remote-as 65030 address-family ipv4 unicast activate neighbor 172.16.1.13 remote-as 65030 address-family ipv4 unicast activate end write memory
verification
show bgp ipv4 unicast summary
spine1
BGP router identifier 10.0.1.13, local AS number 65030 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd 172.16.1.10 4 65033 142 141 0 0 02:13:29 3 172.16.1.2 4 65032 152 163 0 0 02:23:07 3 Total number of neighbors 2 Total number of neighbors established 2
spine2
BGP router identifier 10.0.1.14, local AS number 65030 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd 172.16.1.14 4 65033 145 144 0 0 02:16:31 3 172.16.1.6 4 65032 160 159 0 0 02:31:48 3 Total number of neighbors 2 Total number of neighbors established 2
leaf1
BGP router identifier 10.0.1.24, local AS number 65032 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd 172.16.1.1 4 65030 156 159 0 0 02:27:13 5 172.16.1.5 4 65030 161 164 0 0 02:32:49 5 Total number of neighbors 2 Total number of neighbors established 2
leaf2
BGP router identifier 10.0.1.25, local AS number 65033 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd 172.16.1.13 4 65030 151 154 0 0 02:22:01 5 172.16.1.9 4 65030 151 154 0 0 02:22:04 5 Total number of neighbors 2 Total number of neighbors established 2
EVPN/VXLAN Configuration
Spine
On both spine's the only thing needed to be configured is EVPN and the configuration is the same some both.
Spine1
router bgp 65030 ! address-family l2vpn evpn ! neighbor 172.16.1.10 ! address-family l2vpn evpn activate ! neighbor 172.16.1.2 ! address-family l2vpn evpn activate
Verification
dell-spine1# show bgp l2vpn evpn summary BGP router identifier 10.0.1.13, local AS number 65030 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd 172.16.1.10 4 65033 9612 9605 0 0 00:00:35 0 172.16.1.2 4 65032 9627 9622 0 0 00:00:33 0
Spine2
router bgp 65030 ! address-family l2vpn evpn ! neighbor 172.16.1.14 ! address-family l2vpn evpn activate ! neighbor 172.16.1.6 ! address-family l2vpn evpn activate
Verification
dell-spine2# show bgp l2vpn evpn summary BGP router identifier 10.0.1.14, local AS number 65030 Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State/PfxRcd 172.16.1.14 4 65033 9612 9605 0 0 00:00:35 0 172.16.1.6 4 65032 9627 9622 0 0 00:00:33 0
Leaf
leaf1
interface Loopback 1 description "VXLAN source IP" ip address 10.10.10.1/32
interface vxlan vtep1 source-ip 10.10.10.1
router bgp 65032 address-family l2vpn evpn advertise-all-vni ! neighbor 172.16.1.1 ! address-family l2vpn evpn activate ! neighbor 172.16.1.5 ! address-family l2vpn evpn activate ip vrf Vrf_codfw ip anycast-mac-address 00:00:00:10:10:10 router bgp 65032 vrf Vrf_codfw router-id 10.0.1.24 ! address-family ipv4 unicast redistribute connected ! address-family l2vpn evpn advertise ipv4 unicast !
interface Vlan4000 description "IRB VLAN" ip vrf forwarding Vrf_codfw ipv6 enable ! interface Vlan2004 description private1-e-codfw ip vrf forwarding Vrf_codfw ip anycast-address 10.192.64.254/22 ! interface vxlan vtep1 map vni 102004 vlan 2004 map vni 404000 vlan 4000 map vni 404000 vrf Vrf_codfw ! interface Ethernet0 description test_srv1 no shutdown switchport access Vlan 2004
Verification
dell-leaf1# show vxlan tunnel Name SIP DIP source operstatus ======= ====== ====== ====== ========
The vxlan tunnel is not formed yet, because the switch hasn't learned any routes from the end devices (servers)
dell-leaf1# show vxlan vlanvnimap VLAN VNI ====== ===== Vlan2004 102004 Vlan4000 404000 Total count : 2 dell-leaf1# show vxlan vrfvnimap VRF VNI ====== ===== Vrf_codfw 404000 Total count : 1
leaf 2
interface Loopback 1 description "VXLAN source IP" ip address 10.10.10.2/32
interface vxlan vtep1 source-ip 10.10.10.2
router bgp 65033 address-family l2vpn evpn advertise-all-vni ! neighbor 172.16.1.9 ! address-family l2vpn evpn activate ! neighbor 172.16.1.13 ! address-family l2vpn evpn activate ip vrf Vrf_codfw ip anycast-mac-address 00:00:00:10:10:10 router bgp 65033 vrf Vrf_codfw router-id 10.0.1.25 ! address-family ipv4 unicast redistribute connected ! address-family l2vpn evpn advertise ipv4 unicast !
interface Vlan4000 description "IRB VLAN" ip vrf forwarding Vrf_codfw ipv6 enable ! interface Vlan2004 description private1-e-codfw ip vrf forwarding Vrf_codfw ip anycast-address 10.192.64.254/22 ! interface vxlan vtep1 map vni 102004 vlan 2004 map vni 404000 vlan 4000 map vni 404000 vrf Vrf_codfw ! interface Ethernet0 description test_srv2 no shutdown switchport access Vlan 2004
Verification
dell-leaf1# show vxlan tunnel Name SIP DIP source operstatus ======= ====== ====== ====== ========
The vxlan tunnel is not formed yet, because the switch hasn't learned any routes from the end devices (servers)
dell-leaf2# show vxlan vlanvnimap VLAN VNI ====== ===== Vlan2004 102004 Vlan4000 404000 Total count : 2 dell-leaf1# show vxlan vrfvnimap VRF VNI ====== ===== Vrf_codfw 404000 Total count : 1
End devices connectivity
I setup up 2 servers, one with IP address 10.192.64.10/22 connected to leaf 1 and the second with IP address 10.192.64.20/22 connected to leaf2. After running a ping test from server one to server 2. I checked the vxlan tunnel on both leaf's switches.
dell-leaf1# show vxlan tunnel Name SIP DIP source operstatus ======= ====== ====== ====== ======== EVPN_10.10.10.2 10.10.10.1 10.10.10.2 EVPN oper_up dell-leaf2# show vxlan tunnel Name SIP DIP source operstatus ======= ====== ====== ====== ======== EVPN_10.10.10.1 10.10.10.2 10.10.10.1 EVPN oper_up
Some routes informations
type 5 routes
dell-leaf2# show bgp l2vpn evpn route type prefix
BGP table version is 19, local router ID is 10.0.1.25
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.0.1.24:5096
* [5]:[0]:[22]:[10.192.64.0]
10.10.10.1 0 65030 65032 ?
RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
*> [5]:[0]:[22]:[10.192.64.0]
10.10.10.1 0 65030 65032 ?
RT:65032:404000 ET:8 Rmac:3c:2c:30:4b:09:03
Route Distinguisher: 10.0.1.25:5096
*> [5]:[0]:[22]:[10.192.64.0]
10.10.10.2 0 32768 ?
ET:8 RT:65033:40400 Rmac:3c:2c:30:4c:81:83
Displayed 2 prefixes (3 paths) (of requested type)