Introduction:
This post covers two crucial Geneve Tunnel dependencies to be aware of within NSX-T network virtualized environments:
- Host Transport Node virtualized guest overlay traffic initiates Geneve tunnels.
- Edge Transport Nodes require active Geneve tunnels to establish BGP peering.
High-Level Concept to Remember:
If the Geneve Bidirectional Forwarding Detection (BFD) tunnels are not up due to misconfiguration or other network issues, the Edge’s failover mechanism triggers causing the Edge Node to drop its BGP neighborships. This failsafe mechanism is in place to ensure that the Tier-0 Gateway does not keep advertising routes to the upstream router via BGP when the Edge’s Geneve Tunnels are down, as it won’t be able to process N-S traffic.
NSX-T Lab Topology to demonstrate Geneve Tunnel Dependencies:
Let’s demonstrate these two concepts in the lab, with a focus on BGP peer state and Gevene tunnel state. The following setup is used to illustrate that without active virtualized workloads, there are no tunnels between Geneve Termination End Points (TEPs) and that without Geneve tunnels, BGP peers are down.
Scenario 1: With guest overlay traffic, Geneve Tunnels are active, and BGP Peers establish:
We have a single powered Virtualized Guest, named VM1, on ESXi host esxcna01-s1, generating overlay traffic:
The Guest is creating overlay traffic to initiate a Geneve Tunnel between Compute and Edge nodes:
CLI from Compute Node: [root@esxcna01-s1:~]nsxdp-cli bfd sessions list Remote Local local_disc remote_disc recvd sent local_state local_diag client flaps 192.168.110.180 192.168.110.183 4988543c 32cf4a00 2863 2728 up No Diagnostic vdl2 1 CLI from Edge Node: nsxtedge01> get bfd-sessions | find Dest_port|Encap|address|State Dest_port : 3784 Encap : null Local_address : 192.168.110.65 Remote_address : 192.168.110.66 State : down Dest_port : 3784 Encap : geneve <--- Geneve Tunnel up to Compute Node Local_address : 192.168.110.180 Remote_address : 192.168.110.183 State : up Dest_port : 3784 Encap : vlan Local_address : 192.168.110.180 Remote_address : 192.168.110.181 State : down
The Edge Node Geneve tunnel is a prerequisite to establish BGP peering:
nsxtedge01> vrf 2
nsxtedge01(tier0_sr)> get bgp neighbor summary
BFD States: NC - Not configured, AC - Activating,DC - Disconnected
AD - Admin down, DW - Down, IN - Init,UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 192.168.100.102 Local AS: 65111
Neighbor AS State Up/DownTime BFD InMsgs OutMsgs InPfx OutPfx
192.168.100.2 65100 Estab 01:17:56 NC 91 82 6 4
169.254.0.130 65111 Activ 00:45:08 NC 1104 1108 0 0
BFD States: NC - Not configured, AC - Activating,DC - Disconnected
AD - Admin down, DW - Down, IN - Init,UP - Up
BGP summary information for VRF default for address-family: ipv6Unicast
Router ID: 192.168.100.102 Local AS: 65111
Neighbor AS State Up/DownTime BFD InMsgs OutMsgs InPfx OutPfx
169.254.0.130 65111 Activ 00:45:08 NC 1104 1108 0 0
nsxtedge01(tier0_sr)> get route bgp
Flags: t0c - Tier0-Connected, t0s - Tier0-Static, B - BGP,
t0n - Tier0-NAT, t1s - Tier1-Static, t1c - Tier1-Connected,
t1n: Tier1-NAT, t1l: Tier1-LB VIP, t1ls: Tier1-LB SNAT,
t1d: Tier1-DNS FORWARDER, > - selected route, * - FIB route
Total number of routes: 5
b > * 192.168.150.0/24 [20/0] via 192.168.100.2, uplink-298, 01:18:53
b > * 0.0.0.0/0 [20/0] via 192.168.100.2, uplink-298, 01:18:53
b > * 10.155.14.0/24 [20/0] via 192.168.100.2, uplink-298, 01:18:53
b > * 192.168.21.0/24 [20/0] via 192.168.100.2, uplink-298, 01:18:53
Scenario 2: With guest overlay traffic, Geneve Tunnels drop, and BGP Peering is down:
Virtualized Guest, VM1, is powered down:
Without overlay traffic, the Geneve Tunnel between Compute and Edge nodes drop:
[root@esxcna01-s1:~] nsxdp-cli bfd sessions list Remote Local local_disc remote_disc recvd sent local_state local_diag client flaps nsxtedge01> get bfd-sessions | find Dest_port|Encap|address|State <--- no Geneve Tunnel to Compute Node Dest_port : 3784 Encap : null Local_address : 192.168.110.65 Remote_address : 192.168.110.66 State : down Dest_port : 3784 Encap : vlan Local_address : 192.168.110.180 Remote_address : 192.168.110.181 State : down
Without an active Edge Node Geneve tunnel, there are no BGP neighbors and no learned BGP routes:
nsxtedge01> vrf 2
nsxtedge01(tier0_sr)> get route bgp
Flags: t0c - Tier0-Connected, t0s - Tier0-Static, B - BGP,
t0n - Tier0-NAT, t1s - Tier1-Static, t1c - Tier1-Connected,
t1n: Tier1-NAT, t1l: Tier1-LB VIP, t1ls: Tier1-LB SNAT,
t1d: Tier1-DNS FORWARDER, t1ipsec: Tier1-IPSec,
selected route, * - FIB route
Total number of routes: 0 <--- no BGP routes from physical
This article covers additional details on Geneve Tunnels, and How to Validate MTU in an NSX-T Environment.
Summary:
Although an NSX-T environment without workload traffic isn’t realistic, this lab does help to reinforce some key concepts in NSX-T operation, including two crucial Geneve Tunnel dependencies.
