Revisiting BGP EVPN VXLAN to the hosts with SR Linux 22.6.3
Featuring Netlab 1.3.1 and FRR 8.3.1
At the beginning of this year I wrote about my SR Linux BGP EVPN adventures, with considerations around underlay and overlay design and the illustrious iBGP-over-eBGP approach. Some readers may have noticed a resemblance to the constellation of Ursa Major — the Big Bear: A reference to our friends at CloudBear, a recent SR Linux customer.
Fast forward to September 2022 and we now have SR Linux 22.6.3 with some features I have been waiting for, like (e)BGP Unnumbered. From my side, I have been working with the open source community to extend support for tools like Netlab (formerly Netsim-tools), Containerlab and FRR to enable sophisticated and advanced network topologies using truly open source tools and components.
New features and changes
The issue of running BGP to Linux hosts using FRR popped up in several discussions. Though technically possible, it can be challenging to configure, and there are many design variations with implications that aren’t always obvious. To enable simple experimentation and quick design iterations, I decided to help out by extending Netlab with VLAN, VXLAN and VRF support for FRR. I also made some changes to enable the use of the latest FRR release (8.3.1) just in case there were bug fixes or new knobs that we needed. Then I refactored SR Linux VLAN support, and finally added support for BGP Unnumbered with ipv6-only next hops (RFC8950) for both SR Linux and FRR.
A basic yet beautiful showcase: VXLAN VRFs with EVPN
To start with a simple topology to show off some of the new features in Netlab 1.3.1 (released in September 2022), check out this YAML file and run the following command:
netlab up -d frr integration/evpn/vxlan-routing-ibgp-over-ebgp-unnumbered.yml
Using a simple text file containing only about 80 lines of YAML, this builds out a working lab configuration showcasing the following features:
- Tenant isolation using VRFs ({h1,h2} and {h3,h4})
- Routed VXLAN overlay encapsulation using EVPN as the control plane, in symmetric IRB mode (using RT5 prefixes and a transit l3vni)
- iBGP-over-eBGP using ipv6-lla links and RFC8950 ipv6 next hops
Multi-vendor interop
Testing multi-vendor interop requires only a minor change:
netlab up -d frr -s nodes.r2.device=srlinux vxlan-routing-ibgp-over-ebgp-unnumbered.yml
This instructs Netlab to use ‘srlinux’ as the device type for R2, instead of ‘frr’. The same ping test continues to work; under the hood, Netlab templates take care of sorting out vendor specific issues (such as variations in EVPN RT syntax and auto-derivation, and various knobs that need to be enabled/configured)
eBGP and VXLAN to the FRR hosts
Going back to my original use case: Unlike last time where I was manually configuring VNIs and other parameters, with this new release we can now do fully intent-based networking (IBN).
To start with, a quick note to point out that the original design using default routes for the frr nodes would have issues in case of link failures. Instead, here we send /32 host routes for each loopback, with ECMP enabled. Since all the frr nodes share the same underlay AS, allowas-in must be enabled for them to accept loopback routes towards other frr nodes.
FRR Route leaking
The VRF route leaking aspects deserve some further elaboration. Details can be found in the README and the source scripts; intent is configured in the topology file, where:
- C1 <-> Internet
- C2 <-> Internet and C3
- C3 <-> C2 (only)
EVPN routes are received from the route reflector, and tagged with “2”. Each VRF then locally leaks these routes to the configured peers; routes tagged as “2” become “3”, otherwise leaked routes are tagged as “1”. In the EVPN export policy (“out”) it is then ensured that only locally connected routes are advertised (and not received EVPN routes, or leaked prefixes from other VRFs).
The Spine Internet VRF and the FRR Internet VRFs are configured in a hub-spoke topology; this ensures that all FRR nodes receive “Internet” routes (most likely a simple default route), but not each other’s endpoint routes. This asymmetric import/export construct is admittedly somewhat uncommon and experimental, and it could probably be solved in different ways using alternative import/export policies; YMMV.
Some nitty gritty details of symmetric IRB EVPN
In the course of this exercise I’ve encountered various details that weren’t obvious to me when I started. After explicitly enabling host-route advertisements on SR Linux — required in case of distributed subnets — I discovered that ARP requests by the destination IRB gateway interface only work when it is configured with an anycast gateway IP. Since anycast gateways aren’t supported yet in Netlab, a ping towards a “silent” host currently won’t work; the host has to generate some IP packet itself first.
Closing thoughts
All in all it took a bit longer than anticipated to gather my thoughts and produce this blog to document my findings and progress. It has very much been a trial-and-error exercise, and Ivan must be thinking I either have lost my mind or was acting out some sort of personal vendetta against him, or something. I’ll admit I can be found verging on the edge of reason at times, but I swear it is all with the very best open source networking intentions.
Until we ping again…
