Довольно популярным запросом у заказчиков стало наращивание существующей фабрики Cisco коммутаторами Maipu. Недавно нам удалось наконец собрать необходимый пул оборудования и разобраться, что может и чего не может оборудование Maipu в таких кейсах. Ранее, кстати, рассказал, как мы расширяем существующую DMVPN-сеть Cisco с помощью Maipu, и про коммутаторы Maipu для сетей ЦОД.

Ограничение, которое мы увидели сразу - невозможность реализовать схему Multisite, которую предлагает Cisco. В этой схеме существует специальная роль коммутаторов -- Border Gateway. Эти коммутаторы терминируют VXLAN-туннели внутри фабрики и создают новые туннели до других дата-центров вот так:

Такой подход позволяет ограничить распространение информации, связанной с работой Control Plane, что позволяет лучше масштабировать сети датацентров и сузить домены отказа. При этом между датацентрами работает и L2, и L3.

Функционал основан на нескольких драфтах RFC, и пока не очень распространён. Что-то подобное умеет делать ещё Arista, но на Maipu данная схема недоступна. Поэтому мы пробуем классический Multipod - полный обмен маршрутами между двумя наборами Leaf/Spine-коммутаторов через EBGP. Другой вариант - объединение фабрик через L3, но сейчас не об этом.

Схема, которую нам удалось собрать из имеющегося оборудования, выглядит так:

Второе заметное ограничение - отсутствие на Maipu возможности распространять Broadcast/Unknown Unicast-трафик с помощью PIM и Multicast. Используем BGP Ingress Replication на обоих подах.

На этом грустная часть истории заканчивается и начинается интересная.

Фабрику настраиваем так же, как и в предыдущей серии: в Underlay OSPF и IBGP. Такой вариант мы чаще всего встречаем у заказчиков, эксплуатирующих фабрику на Cisco. Хосты сразу подключаем к двум Leaf-коммутаторам, это VPC и MLAG соответственно.

Между Spine-коммутаторами, где у нас, собственно, и происходит соединение двух фабрик, настраиваем EBGP. Для того, чтобы VXLAN-туннели корректно строились от одного Leaf-коммутатора до другого, сохраняем значение атрибута Next-Hop в передаваемых EVPN-маршрутах.

Конфиги Leaf-коммутаторов Cisco Nexus:

N9k-Leaf1

N9k-Leaf2

cfs eth distribute

nv overlay evpn

feature ospf

feature bgp

feature pim

feature fabric forwarding

feature interface-vlan

feature vn-segment-vlan-based

feature hsrp

feature lacp

feature vpc

feature lldp

feature bfd

feature nv overlay

fabric forwarding anycast-gateway-mac 0001.0001.0100

vlan 1,41-42,999

vlan 41

vn-segment 100

vlan 42

vn-segment 200

vlan 999

vn-segment 1

route-map DIRECT-BGP-RM permit 10

match tag 12345

vrf context Keepalive

vrf context vxlan-100

vni 1

rd 100:42

address-family ipv4 unicast

route-target import 100:2

route-target import 100:2 evpn

route-target export 100:2

route-target export 100:2 evpn

vpc domain 1

peer-keepalive destination 10.253.1.5 source 10.253.1.4 vrf Keepalive

interface Vlan41

no shutdown

vrf member vxlan-100

ip address 172.16.129.1/24

fabric forwarding mode anycast-gateway

interface Vlan42

no shutdown

vrf member vxlan-100

ip address 172.16.130.1/24

fabric forwarding mode anycast-gateway

interface Vlan999

no shutdown

vrf member vxlan-100

ip forward

interface port-channel30

switchport

switchport mode trunk

switchport trunk allowed vlan 41-42

vpc 1

interface port-channel54

switchport

switchport mode trunk

spanning-tree port type network

vpc peer-link

interface nve1

no shutdown

host-reachability protocol bgp

source-interface loopback101

member vni 1 associate-vrf

member vni 100

ingress-replication protocol bgp

member vni 200

ingress-replication protocol bgp

interface Ethernet1/1

switchport

switchport mode trunk

switchport trunk allowed vlan 41-42

channel-group 30 mode active

no shutdown

interface Ethernet1/48

description -= Peer Keepalive Link =-

vrf member Keepalive

ip address 10.253.1.4/31

no shutdown

interface Ethernet1/53

description -= Spine =-

ip address 10.253.1.1/31

ip ospf network point-to-point

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

no shutdown

interface Ethernet1/54

description -= Peer Link =-

switchport

switchport mode trunk

channel-group 54 mode active

no shutdown

interface loopback0

ip address 10.253.127.1/32

ip router ospf 1 area 0.0.0.0

interface loopback101

ip address 10.253.127.253/32

ip address 10.253.127.40/32 secondary

ip router ospf 1 area 0.0.0.0

router ospf 1

router-id 10.253.127.1

router bgp 200

address-family l2vpn evpn

neighbor 10.253.127.3

remote-as 200

update-source loopback0

address-family ipv4 unicast

advertisement-interval 1

address-family l2vpn evpn

send-community

send-community extended

vrf vxlan-100

address-family ipv4 unicast

network 172.16.129.0/24

network 172.16.130.0/24

advertise l2vpn evpn

redistribute direct route-map DIRECT-BGP-RM

evpn

vni 100 l2

rd 100:40

route-target import 100:1

route-target export 100:1

vni 200 l2

rd 200:40

route-target import 200:1

route-target export 200:1

cfs eth distribute

nv overlay evpn

feature ospf

feature bgp

feature pim

feature fabric forwarding

feature interface-vlan

feature vn-segment-vlan-based

feature hsrp

feature lacp

feature vpc

feature lldp

feature bfd

feature nv overlay

fabric forwarding anycast-gateway-mac 0001.0001.0100

vlan 1,41-42,999

vlan 41

vn-segment 100

vlan 42

vn-segment 200

vlan 999

vn-segment 1

route-map DIRECT-BGP-RM permit 10

match tag 12345

vrf context Keepalive

vrf context vxlan-100

vni 1

rd 100:42

address-family ipv4 unicast

route-target import 100:2

route-target import 100:2 evpn

route-target export 100:2

route-target export 100:2 evpn

vpc domain 1

peer-keepalive destination 10.253.1.4 source 10.253.1.5 vrf Keepalive

interface Vlan41

no shutdown

vrf member vxlan-100

ip address 172.16.129.1/24

fabric forwarding mode anycast-gateway

interface Vlan42

no shutdown

vrf member vxlan-100

ip address 172.16.130.1/24

fabric forwarding mode anycast-gateway

interface Vlan999

no shutdown

vrf member vxlan-100

ip forward

interface port-channel30

switchport

switchport mode trunk

switchport trunk allowed vlan 41-42

vpc 1

interface port-channel54

switchport

switchport mode trunk

spanning-tree port type network

vpc peer-link

interface nve1

no shutdown

host-reachability protocol bgp

source-interface loopback101

member vni 1 associate-vrf

member vni 100

ingress-replication protocol bgp

member vni 200

ingress-replication protocol bgp

interface Ethernet1/1

switchport

switchport mode trunk

switchport trunk allowed vlan 41-42

channel-group 30 mode active

no shutdown

interface Ethernet1/48

description -= Peer Keepalive Link =-

vrf member Keepalive

ip address 10.253.1.5/31

no shutdown

interface Ethernet1/53

description -= Spine =-

ip address 10.253.1.3/31

ip ospf network point-to-point

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

no shutdown

interface Ethernet1/54

description -= Peer Link =-

switchport

switchport mode trunk

channel-group 54 mode active

no shutdown

interface loopback0

ip address 10.253.127.2/32

ip router ospf 1 area 0.0.0.0

interface loopback101

ip address 10.253.127.254/32

ip address 10.253.127.40/32 secondary

ip router ospf 1 area 0.0.0.0

router ospf 1

router-id 10.253.127.2

router bgp 200

address-family l2vpn evpn

neighbor 10.253.127.3

remote-as 200

update-source loopback0

address-family ipv4 unicast

advertisement-interval 1

address-family l2vpn evpn

send-community

send-community extended

vrf vxlan-100

address-family ipv4 unicast

network 172.16.129.0/24

network 172.16.130.0/24

advertise l2vpn evpn

redistribute direct route-map DIRECT-BGP-RM

evpn

vni 100 l2

rd 100:40

route-target import 100:1

route-target export 100:1

vni 200 l2

rd 200:40

route-target import 200:1

route-target export 200:1

Spine Nexus:

N9k-Spine

nv overlay evpn

feature ospf

feature bgp

feature pim

feature fabric forwarding

feature interface-vlan

feature vn-segment-vlan-based

feature hsrp

feature lacp

feature lldp

feature bfd

feature nv overlay

vlan 1

route-map BGP-OUT-RM permit 10

set ip next-hop unchanged

vrf context vxlan-100

vni 1

rd 100:4

address-family ipv4 unicast

route-target import 100:2

route-target export 100:2

interface Ethernet1/48

ip address 11.0.0.2/24

ip router ospf 1 area 0.0.0.0

no shutdown

interface Ethernet1/49

description -= External Port =-

ip address 10.254.100.3/24

ip ospf mtu-ignore

ip router ospf 1 area 0.0.0.0

no shutdown

interface Ethernet1/53

description -= Leaf 1 =-

ip address 10.253.1.0/31

ip ospf network point-to-point

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

no shutdown

interface Ethernet1/54

description -= Leaf 2 =-

ip address 10.253.1.2/31

ip ospf network point-to-point

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

no shutdown

interface loopback0

ip address 10.253.127.3/32

ip router ospf 1 area 0.0.0.0

router ospf 1

router-id 10.253.127.3

router bgp 200

neighbor 10.253.127.1

remote-as 200

update-source loopback0

address-family ipv4 unicast

advertisement-interval 1

address-family l2vpn evpn

send-community

send-community extended

neighbor 10.253.127.2

remote-as 200

update-source loopback0

address-family ipv4 unicast

advertisement-interval 1

address-family l2vpn evpn

send-community

send-community extended

neighbor 10.254.127.1

remote-as 100

update-source loopback0

ebgp-multihop 5

address-family ipv4 unicast

advertisement-interval 1

address-family l2vpn evpn

send-community

send-community extended

route-map BGP-OUT-RM out

neighbor 10.254.127.2

remote-as 100

update-source loopback0

ebgp-multihop 5

address-family ipv4 unicast

advertisement-interval 1

address-family l2vpn evpn

send-community

send-community extended

route-map BGP-OUT-RM out

vrf vxlan-100

address-family ipv4 unicast

advertise l2vpn evpn

evpn

vni 100 l2

rd 100:3

route-target import 100:1

route-target export 100:1

vni 200 l2

rd 200:3

route-target import 200:1

route-target export 200:1

Leaf Maipu:

Maipu-L11

Maipu-L12

ip vrf 100

rd 100:2

l3vnid 1

address-family evpn

route-target import 100:2 ipv4

route-target export 100:2 ipv4

exit

exit

link-aggregation 64 mode lacp

link-aggregation 250 mode lacp

vlan 1

exit

vlan 11-13

mlag domain 1

node id 1

node role-priority 50

role preempt

system-mac 0001.7a95.000b

keepalive ip destination 192.168.168.41 source 192.168.168.40

exit

vxlan 100

vxlan vnid 100

address-family evpn

rd 100:10

route-target import 100:1

route-target export 100:1

exit

exit

vxlan 200

vxlan vnid 200

address-family evpn

rd 200:10

route-target import 200:1

route-target export 200:1

exit

exit

interface tengigabitethernet0/48

link-aggregation 64 active

exit

interface 100ge0/49

no switchport

ip address 10.254.1.1 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/50

no switchport

ip address 10.254.2.1 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/55

fec mode rs auto

link-aggregation 250 active

exit

interface 100ge0/56

fec mode rs auto

link-aggregation 250 active

exit

!end

interface link-aggregation64

switchport mode trunk

switchport trunk allowed vlan add 1,11-12

switchport trunk pvid vlan 1

vxlan 100 encapsulation vlan 11

vxlan 200 encapsulation vlan 12

mlag group 1

exit

interface link-aggregation250

switchport mode trunk

switchport trunk allowed vlan add 11-13

switchport trunk pvid vlan 1

mlag peer-link

exit

interface dc0

ip address 192.168.168.40 255.255.0.0

exit

interface loopback1

description Router-ID

ip address 10.254.127.11 255.255.255.255

exit

interface loopback2

description VTEP

ip address 10.254.128.11 255.255.255.255

ip address 10.254.128.10 255.255.255.255 secondary

exit

interface loopback100

ip vrf forwarding 100

ip address 172.16.250.11 255.255.255.255

exit

interface vxlan100

ip vrf forwarding 100

vxlan distribute-gateway

ip address 172.16.129.1 255.255.255.0

mac-address 0001.0001.0100

exit

interface vxlan200

ip vrf forwarding 100

vxlan distribute-gateway

ip address 172.16.130.1 255.255.255.0

mac-address 0001.0001.0100

exit

interface nve1

source 10.254.128.10

mac-address 0088.9999.1112

vxlan 100,200 ingress-replication protocol bgp

exit

router ospf 100

router-id 10.254.127.11

bfd all-interfaces

network 10.254.1.1 0.0.0.0 area 0

network 10.254.2.1 0.0.0.0 area 0

network 10.254.127.11 0.0.0.0 area 0

network 10.254.128.10 0.0.0.0 area 0

maximum-paths 8

exit

router bgp 100

no auto-summary

no synchronization

bgp router-id 10.254.127.11

neighbor spine peer-group

neighbor spine remote-as 100

neighbor spine update-source loopback1

neighbor 10.254.127.1 peer-group spine

neighbor 10.254.127.2 peer-group spine

maximum-paths ibgp 8

address-family l2vpn evpn

neighbor spine activate

neighbor spine send-community both

neighbor 10.254.127.1 activate

neighbor 10.254.127.1 send-community both

neighbor 10.254.127.2 activate

neighbor 10.254.127.2 send-community both

exit-address-family

address-family ipv4 vrf 100

advertise-l2vpn-evpn

network 172.16.129.0 255.255.255.0

network 172.16.130.0 255.255.255.0

network 172.16.250.11 255.255.255.255

network 172.16.250.12 255.255.255.255

exit-address-family

exit

ip vrf 100

rd 100:2

l3vnid 1

address-family evpn

route-target import 100:2 ipv4

route-target export 100:2 ipv4

exit

exit

link-aggregation 64 mode lacp

link-aggregation 250 mode lacp

vlan 1

exit

vlan 11-13

mlag domain 1

node id 2

system-mac 0001.7a95.000b

keepalive ip destination 192.168.168.40 source 192.168.168.41

exit

vxlan 100

vxlan vnid 100

address-family evpn

rd 100:10

route-target import 100:1

route-target export 100:1

exit

exit

vxlan 200

vxlan vnid 200

address-family evpn

rd 200:10

route-target import 200:1

route-target export 200:1

exit

exit

interface tengigabitethernet0/48

link-aggregation 64 active

exit

interface 100ge0/49

no switchport

ip address 10.254.1.3 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/50

no switchport

ip address 10.254.2.3 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/55

fec mode rs auto

link-aggregation 250 active

exit

interface 100ge0/56

fec mode rs auto

link-aggregation 250 active

exit

!end

interface link-aggregation64

switchport mode trunk

switchport trunk allowed vlan add 1,11-12

switchport trunk pvid vlan 1

vxlan 100 encapsulation vlan 11

vxlan 200 encapsulation vlan 12

mlag group 1

exit

interface link-aggregation250

switchport mode trunk

switchport trunk allowed vlan add 11-13

switchport trunk pvid vlan 1

mlag peer-link

exit

interface dc0

ip address 192.168.168.41 255.255.0.0

exit

interface loopback1

description Router-ID

ip address 10.254.127.12 255.255.255.255

exit

interface loopback2

description VTEP

ip address 10.254.128.12 255.255.255.255

ip address 10.254.128.10 255.255.255.255 secondary

exit

interface loopback100

ip vrf forwarding 100

ip address 172.16.250.12 255.255.255.255

exit

interface vxlan100

ip vrf forwarding 100

vxlan distribute-gateway

ip address 172.16.129.1 255.255.255.0

mac-address 0001.0001.0100

exit

interface vxlan200

ip vrf forwarding 100

vxlan distribute-gateway

ip address 172.16.130.1 255.255.255.0

mac-address 0001.0001.0100

exit

interface nve1

source 10.254.128.10

mac-address 0088.9999.1112

vxlan 100,200 ingress-replication protocol bgp

exit

router ospf 100

router-id 10.254.127.12

bfd all-interfaces

network 10.254.1.3 0.0.0.0 area 0

network 10.254.2.3 0.0.0.0 area 0

network 10.254.127.12 0.0.0.0 area 0

network 10.254.128.10 0.0.0.0 area 0

maximum-paths 8

exit

router bgp 100

no auto-summary

no synchronization

bgp router-id 10.254.127.12

neighbor spine peer-group

neighbor spine remote-as 100

neighbor spine update-source loopback1

neighbor 10.254.127.1 peer-group spine

neighbor 10.254.127.2 peer-group spine

maximum-paths ibgp 8

address-family l2vpn evpn

neighbor spine activate

neighbor spine send-community both

neighbor 10.254.127.1 activate

neighbor 10.254.127.1 send-community both

neighbor 10.254.127.2 activate

neighbor 10.254.127.2 send-community both

exit-address-family

address-family ipv4 vrf 100

advertise-l2vpn-evpn

network 172.16.129.0 255.255.255.0

network 172.16.130.0 255.255.255.0

network 172.16.250.11 255.255.255.255

network 172.16.250.12 255.255.255.255

exit-address-family

exit

Spine Maipu:

Maipu-S1

Maipu-S2

ip vrf 100

rd 100:3

l3vnid 1

address-family evpn

route-target import 100:2 ipv4

route-target export 100:2 ipv4

exit

exit

vxlan 100

vxlan vnid 100

address-family evpn

rd 100:1

route-target import 100:1

route-target export 100:1

exit

exit

vxlan 200

vxlan vnid 200

address-family evpn

rd 200:1

route-target import 200:1

route-target export 200:1

exit

exit

interface 100ge0/1

no switchport

ip address 10.254.1.0 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/2

no switchport

ip address 10.254.1.2 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/6

no switchport

ip address 10.254.1.4 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/10

no switchport

ip address 10.254.1.6 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/14

no switchport

ip address 10.254.100.1 255.255.255.0

ip ospf mtu-ignore

exit

!end

interface loopback1

description Router-ID

ip address 10.254.127.1 255.255.255.255

exit

router ospf 100

router-id 10.254.127.1

bfd all-interfaces

network 10.254.1.0 0.0.0.0 area 0

network 10.254.1.2 0.0.0.0 area 0

network 10.254.1.4 0.0.0.0 area 0

network 10.254.1.6 0.0.0.0 area 0

network 10.254.100.0 0.0.0.255 area 0

network 10.254.127.1 0.0.0.0 area 0

exit

router bgp 100

no auto-summary

no synchronization

neighbor leaf peer-group

neighbor leaf remote-as 100

neighbor leaf update-source loopback1

neighbor leaf advertisement-interval 0

neighbor 10.253.127.3 remote-as 200

neighbor 10.253.127.3 ebgp-multihop 10

neighbor 10.253.127.3 update-source loopback1

neighbor 10.253.127.3 advertisement-interval 0

neighbor 10.253.127.3 attribute-unchanged

neighbor 10.254.127.11 peer-group leaf

neighbor 10.254.127.12 peer-group leaf

neighbor 10.254.127.21 peer-group leaf

address-family l2vpn evpn

neighbor 10.253.127.3 activate

neighbor 10.253.127.3 send-community both

neighbor 10.253.127.3 attribute-unchanged

neighbor 10.254.127.11 activate

neighbor 10.254.127.11 route-reflector-client

neighbor 10.254.127.11 send-community both

neighbor 10.254.127.12 activate

neighbor 10.254.127.12 route-reflector-client

neighbor 10.254.127.12 send-community both

neighbor 10.254.127.21 activate

neighbor 10.254.127.21 route-reflector-client

neighbor 10.254.127.21 send-community both

exit-address-family

address-family ipv4 vrf 100

advertise-l2vpn-evpn

exit-address-family

exit

ip vrf 100

rd 100:3

l3vnid 1

address-family evpn

route-target import 100:2 ipv4

route-target export 100:2 ipv4

exit

exit

vxlan 100

vxlan vnid 100

address-family evpn

rd 100:2

route-target import 100:1

route-target export 100:1

exit

exit

vxlan 200

vxlan vnid 200

address-family evpn

rd 200:2

route-target import 200:1

route-target export 200:1

exit

exit

interface 100ge0/1

no switchport

ip address 10.254.2.0 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/2

no switchport

ip address 10.254.2.2 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/6

no switchport

ip address 10.254.2.4 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/10

no switchport

ip address 10.254.2.6 255.255.255.254

no ip redirects

ip ospf network point-to-point

ip ospf bfd

ip mtu exception forward

exit

interface 100ge0/14

no switchport

ip address 10.254.100.2 255.255.255.0

ip ospf mtu-ignore

exit

!end

interface loopback1

description Router-ID

ip address 10.254.127.2 255.255.255.255

exit

router ospf 100

router-id 10.254.127.2

bfd all-interfaces

network 2.2.2.2 0.0.0.0 area 0

network 10.254.2.0 0.0.0.0 area 0

network 10.254.2.2 0.0.0.0 area 0

network 10.254.2.4 0.0.0.0 area 0

network 10.254.100.0 0.0.0.255 area 0

network 10.254.127.2 0.0.0.0 area 0

exit

router bgp 100

no auto-summary

no synchronization

neighbor leaf peer-group

neighbor leaf remote-as 100

neighbor leaf update-source loopback1

neighbor leaf advertisement-interval 0

neighbor 10.253.127.3 remote-as 200

neighbor 10.253.127.3 ebgp-multihop 10

neighbor 10.253.127.3 update-source loopback1

neighbor 10.253.127.3 advertisement-interval 0

neighbor 10.253.127.3 attribute-unchanged

neighbor 10.254.127.11 peer-group leaf

neighbor 10.254.127.12 peer-group leaf

neighbor 10.254.127.21 peer-group leaf

address-family l2vpn evpn

neighbor 10.253.127.3 activate

neighbor 10.253.127.3 send-community both

neighbor 10.253.127.3 attribute-unchanged

neighbor 10.254.127.11 activate

neighbor 10.254.127.11 route-reflector-client

neighbor 10.254.127.11 send-community both

neighbor 10.254.127.12 activate

neighbor 10.254.127.12 route-reflector-client

neighbor 10.254.127.12 send-community both

neighbor 10.254.127.21 activate

neighbor 10.254.127.21 route-reflector-client

neighbor 10.254.127.21 send-community both

exit-address-family

address-family ipv4 vrf 100

advertise-l2vpn-evpn

exit-address-family

exit

Результат

Хосты друг друга пингуют, прочий трафик тоже ходит. Честно говоря, в этом мы особенно и не сомневались. Опасения вызывало то, как поймут друг друга фабрики, в которых присутствует с одной стороны VPC, а с другой Multi-Chassis LAG. Подёргав интерфейсы с каждой из сторон, убедились в том, что всё работает корректно. Впрочем, на случай проблем с MLAG, на коммутаторах Maipu серии ЦОД доступно стекирование.

Вот так выглядят маршруты на Nexus:

N9k-Leaf1# sh bgp l2vpn evpn

BGP routing table information for VRF default, address family L2VPN EVPN

BGP table version is 1286, Local Router ID is 10.253.127.1

Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best

Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected

Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

Network Next Hop Metric LocPrf Weight Path

Route Distinguisher: 100:1

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216

10.254.128.20 0 100 0 100 i

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.129.1]/272

10.254.128.20 0 100 0 100 i

*>i[3]:[0]:[32]:[10.254.128.20]/88

10.254.128.20 0 100 0 100 i

Route Distinguisher: 100:10

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216

10.254.128.10 0 100 0 100 i

*>i[2]:[0]:[0]:[48]:[dc4a.3e3b.cbf8]:[0]:[0.0.0.0]/216

10.254.128.10 0 100 0 100 i

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.129.1]/272

10.254.128.10 0 100 0 100 i

*>i[2]:[0]:[0]:[48]:[dc4a.3e3b.cbf8]:[32]:[172.16.129.11]/272

10.254.128.10 0 100 0 100 i

*>i[3]:[0]:[32]:[10.254.128.10]/88

10.254.128.10 0 100 0 100 i

Route Distinguisher: 100:40 (L2VNI 100)

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216

10.254.128.10 0 100 0 100 i

* i 10.254.128.20 0 100 0 100 i

*>i[2]:[0]:[0]:[48]:[dc4a.3e3b.cbf8]:[0]:[0.0.0.0]/216

10.254.128.10 0 100 0 100 i

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.129.1]/272

10.254.128.10 0 100 0 100 i

* i 10.254.128.20 0 100 0 100 i

*>i[2]:[0]:[0]:[48]:[dc4a.3e3b.cbf8]:[32]:[172.16.129.11]/272

10.254.128.10 0 100 0 100 i

*>l[3]:[0]:[32]:[10.253.127.40]/88

10.253.127.40 100 32768 i

*>i[3]:[0]:[32]:[10.254.128.10]/88

10.254.128.10 0 100 0 100 i

*>i[3]:[0]:[32]:[10.254.128.20]/88

10.254.128.20 0 100 0 100 i

Route Distinguisher: 200:10

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216

10.254.128.10 0 100 0 100 i

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.130.1]/272

10.254.128.10 0 100 0 100 i

*>i[3]:[0]:[32]:[10.254.128.10]/88

10.254.128.10 0 100 0 100 i

Route Distinguisher: 200:40 (L2VNI 200)

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216

10.254.128.10 0 100 0 100 i

*>l[2]:[0]:[0]:[48]:[dc4a.3e3b.cbe9]:[0]:[0.0.0.0]/216

10.253.127.40 100 32768 i

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.130.1]/272

10.254.128.10 0 100 0 100 i

*>l[2]:[0]:[0]:[48]:[dc4a.3e3b.cbe9]:[32]:[172.16.130.11]/272

10.253.127.40 100 32768 i

*>l[3]:[0]:[32]:[10.253.127.40]/88

10.253.127.40 100 32768 i

*>i[3]:[0]:[32]:[10.254.128.10]/88

10.254.128.10 0 100 0 100 i

Route Distinguisher: 100:42 (L3VNI 1)

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[0]:[0.0.0.0]/216

10.254.128.10 0 100 0 100 i

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.129.1]/272

10.254.128.10 0 100 0 100 i

* i 10.254.128.20 0 100 0 100 i

*>i[2]:[0]:[0]:[48]:[0001.0001.0100]:[32]:[172.16.130.1]/272

10.254.128.10 0 100 0 100 i

*>i[2]:[0]:[0]:[48]:[dc4a.3e3b.cbf8]:[32]:[172.16.129.11]/272

10.254.128.10 0 100 0 100 i

*>l[5]:[0]:[0]:[24]:[172.16.129.0]/224

10.253.127.40 100 32768 i

*>l[5]:[0]:[0]:[24]:[172.16.130.0]/224

10.253.127.40 100 32768 i

А вот вывод с Maipu:

L11#show bgp l2vpn evpn all all-type

BGP local router ID is 10.254.127.11

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

S Stale

Origin codes: i - IGP, e - EGP, ? - incomplete

EVPN Information for Route Distinguisher:100:1

MAC/IP Advertisement Routes:

Network(ETID:MAC:IP) Next Hop Metric LocPrf Weight Path

[B]*>i0:48:0001.0001.0100:0:0.0.0.0/96 10.254.128.20 0 100 0 i

[B]* i 10.254.128.20 0 100 0 i

[B]*>i0:48:0001.0001.0100:32:172.16.129.1/128 10.254.128.20 0 100 0 i

[B]* i 10.254.128.20 0 100 0 i

Inclusive Multicast Ethernet Tag Routes:

Network(Originating IP Addr) Next Hop Metric LocPrf Weight Path

[B]*>i0:32:10.254.128.20/72 10.254.128.20 0 100 0 i

[B]* i 10.254.128.20 0 100 0 i

EVPN Information for Route Distinguisher:100:2

MAC/IP Advertisement Routes:

Network(ETID:MAC:IP) Next Hop Metric LocPrf Weight Path

[B]*> 0:48:0001.0001.0100:32:172.16.129.1/128 0.0.0.0 0 32768 i

[B]* i 10.254.128.20 0 100 0 i

[B]*> 0:48:dc4a.3e3b.cbf8:32:172.16.129.11/128 0.0.0.0 0 32768 i

[B]*> 0:48:0001.0001.0100:32:172.16.130.1/128 0.0.0.0 0 32768 i

[B]*>i0:48:dc4a.3e3b.cbe9:32:172.16.130.11/128 10.253.127.40 0 100 0 200 i

IP Prefix Routes:

Network(ETID:IP) Next Hop Metric LocPrf Weight Path

[B]*> 0:24:172.16.129.0/72 0.0.0.0 0 32768 i

[B]*> 0:24:172.16.130.0/72 0.0.0.0 0 32768 i

EVPN Information for Route Distinguisher:100:10

MAC/IP Advertisement Routes:

Network(ETID:MAC:IP) Next Hop Metric LocPrf Weight Path

[B]*> 0:48:0001.0001.0100:0:0.0.0.0/96 0.0.0.0 0 32768 i

[B]* i 10.254.128.20 0 100 0 i

[B]*> 0:48:dc4a.3e3b.cbf8:0:0.0.0.0/96 0.0.0.0 0 32768 i

[B]*> 0:48:0001.0001.0100:32:172.16.129.1/128 0.0.0.0 0 32768 i

[B]* i 10.254.128.20 0 100 0 i

[B]*> 0:48:dc4a.3e3b.cbf8:32:172.16.129.11/128 0.0.0.0 0 32768 i

Inclusive Multicast Ethernet Tag Routes:

Network(Originating IP Addr) Next Hop Metric LocPrf Weight Path

[B]*>i0:32:10.253.127.40/72 10.253.127.40 0 100 0 200 i

[B]*> 0:32:10.254.128.10/72 0.0.0.0 0 32768 i

[B]*>i0:32:10.254.128.20/72 10.254.128.20 0 100 0 i

EVPN Information for Route Distinguisher:100:40

Inclusive Multicast Ethernet Tag Routes:

Network(Originating IP Addr) Next Hop Metric LocPrf Weight Path

[B]* i0:32:10.253.127.40/72 10.253.127.40 0 100 0 200 i

[B]*>i 10.253.127.40 0 100 0 200 i

EVPN Information for Route Distinguisher:200:10

MAC/IP Advertisement Routes:

Network(ETID:MAC:IP) Next Hop Metric LocPrf Weight Path

[B]*> 0:48:0001.0001.0100:0:0.0.0.0/96 0.0.0.0 0 32768 i

[B]*>i0:48:dc4a.3e3b.cbe9:0:0.0.0.0/96 10.253.127.40 0 100 0 200 i

[B]* 0.0.0.0 0 32768 i

[B]*>i0:48:dc4a.3e3b.cbf8:0:0.0.0.0/96 10.253.127.40 0 100 0 200 i

[B]*> 0:48:0001.0001.0100:32:172.16.130.1/128 0.0.0.0 0 32768 i

[B]*>i0:48:dc4a.3e3b.cbe9:32:172.16.130.11/128 10.253.127.40 0 100 0 200 i

Inclusive Multicast Ethernet Tag Routes:

Network(Originating IP Addr) Next Hop Metric LocPrf Weight Path

[B]*>i0:32:10.253.127.40/72 10.253.127.40 0 100 0 200 i

[B]*> 0:32:10.254.128.10/72 0.0.0.0 0 32768 i

EVPN Information for Route Distinguisher:200:40

MAC/IP Advertisement Routes:

Network(ETID:MAC:IP) Next Hop Metric LocPrf Weight Path

[B]*>i0:48:dc4a.3e3b.cbe9:0:0.0.0.0/96 10.253.127.40 0 100 0 200 i

[B]* i 10.253.127.40 0 100 0 200 i

[B]* i0:48:dc4a.3e3b.cbf8:0:0.0.0.0/96 10.253.127.40 0 100 0 200 i

[B]*>i 10.253.127.40 0 100 0 200 i

[B]*>i0:48:dc4a.3e3b.cbe9:32:172.16.130.11/128 10.253.127.40 0 100 0 200 i

[B]* i 10.253.127.40 0 100 0 200 i

Inclusive Multicast Ethernet Tag Routes:

Network(Originating IP Addr) Next Hop Metric LocPrf Weight Path

[B]* i0:32:10.253.127.40/72 10.253.127.40 0 100 0 200 i

[B]*>i 10.253.127.40 0 100 0 200 i

Проверили доступность хостов и по L2, при помещении их в один VXLAN-сегмент.

Выводы

Главный вывод - мы не напоролись ни на какой подводный камень. Получили работающий Asymmetric Routing & Bridging. Фабрику Cisco можно нарастить подами Maipu, пусть и при определённых ограничениях.

Хорошим продолжением было бы тестирование такой схемы на пилотном сегменте реальной сети так, чтобы подключить побольше хостов разных типов, получить много MAC-адресов, маршрутов и реальный трафик.

На момент написания поста мы уже получили в лабораторию набор оборудования Maipu, включающий коммутаторы линейки ЦОД и готовы тестировать более конкретные, жизненные кейсы для наших заказчиков.

Кроме того, 18 октября у нас пройдёт онлайн-митап, на котором мы расскажем о доступных сегодня в России альтернативах и на примере Maipu продемонстрируем, как они справляются с типовыми кейсами заказчиков.

Связываем VXLAN-фабрики на Maipu и Cisco