Introduction:
NSX-T TEP IP address misconfiguration is one of the most common issues associated with new NSX-T deployments. It can lead to downed GENEVE tunnels between ESXi hosts and NSX-T Edges, in collapsed Compute/Edge Cluster topologies. In this scenario, NSX-T backed virtualized guests can reach each other, but not the physical environment. In this article, I will do my best to describe the problem scenario and some possible solutions. Let’s see if a picture is worth a 1000 words;)
Failure Scenario
This failure scenario illustrates a common setup issue in new deployments, where the Edge TEP to Host TEP GENEVE tunnel can’t form since traffic isn’t traversing the ESXi IOChain for GENEVE encapsulation and decapsulation. The host does not currently have the ability to internally hairpin GENEVE traffic and perform the required GENEVE encapsulation or decapsulation.
Edge and host management interfaces can communicate since there is no GENEVE encapsulation or decapsulation required.
The following diagram illustrates where ESXi IOChain GENEVE encapsulation/decapsulation occurs.
This article covers three possible solutions, with a focus on NSX-T TEP IP addressing.
Solution 1: Separate Edge and Compute Clusters
- Separate Edge and Compute clusters means that Compute and Edge TEP interfaces do not coexist on the same host.
- Host TEPand NSX Edge TEP can be in the same or different subnets.
- This solution may not be a viable option in scenarios where hardware is limited.
- The N-VDS + VSS/VDS MTU needs to support the GENEVE frame size, say 1600 or 9000 bytes.
Solution 2: Introduce a VDS for the Edge
- The NSX-T Edge VM connects to VSS (Virtual Standard Switch) or VDS (Virtual Distributed Switch) port groups where the VSS/VDS consume separate pNICs (physical NICs) on the host.
- Host TEP and NSX Edge TEP can be in the same or different subnets.
- This solution requires additional pNICs, which may not be available in two pNIC blade deployments.
- The N-VDS + VSS/VDS MTU needs to support the GENEVE frame size, say 1600 or 9000 bytes.
Solution 3: Introduce a new Edge TEP subnet
- Host TEP and NSX Edge TEP must be in different subnets.
- It is not enough to separate Host TEP and NSX Edge TEP IPs into separate IP ranges within the same subnet.
- for example, this will not work, a TEP Subnet of 192.168.110.0/24
- where the Host TEP range is 192.168.110.1-.127/24, and the Host TEP range is 192.168.110.129-.154 /24
- since these are just separate IP ranges within the same subnet.
- The N-VDS + VSS/VDS MTU needs to support the GENEVE frame size, say 1600 or 9000 bytes.
- This solution requires additional configuration on the physical network:
- this requires L3 routing on physical between Host and Edge TEP subnets
- L2 VLANs MTU needs to support the GENEVE frame size
- L3 physical router interfaces MTU needs to support the GENEVE frame size
Additional Resources:
Here are two links for some additional reading:
Flexible deployment options for NSX-T Data Center Edge VM
From the VMware NSX-T Data Center Installation Guide, NSX Edge Networking Setup:
Summary
I hope that this helps clarify some different NSX-T network topology options and their impact on the TEP IP address configuration. For additional reading, you may find this article helpful in connecting NSX-T virtual to physical environments.
