Introduction:
NSX-T Data Center Edges carry all traffic in and out of the Software-Defined Data Center network, providing connectivity between virtualized and physical environments.
The goal of this blog post is to review some issues that result in an inability for a Tier-0 gateway to ping it’s directly connected physical peer IP.
NSX-T introduces a number of additional complexities to the mix, which can make connecting a Tier-0 Gateway to the physical environment challenging. Some of these include:
- two Edge form factors, Virtual Machine (covered in this post) and Bare Metal
- the concept of an Edge acting as a Gateway container
- the introduction of the NSX-T Virtual Distributed Switch (N-VDS)
- the support for multiple topologies
- the Edge VM has an N-VDS, which can be shared by the host
- the ESXi host may share an N-VDS with the Edge, or may have it’s own N-VDS, VDS, or VSS though which Edge traffic must travel to reach physical
- the ability to deploy the Edge from an OVA or UI
Edge VM Network Topology Overview:
Amit Aneja breaks down flexible deployment options for NSX-T Data Center Edges, which serves as an excellent reference and is a prerequisite to configuring Tier-0 connectivity to physical.
The NSX-T Data Center Installation Guide has the best depiction I’ve been able to find of what is becoming a more familiar Edge deployment, NSX-T Edge VM deployed on a compute host using N-VDS Logical switches in Collapsed Compute and Edge Cluster, where the NSX Edge VM shares the same N-VDS as the host. Here is the original image with some markups shown in red:
In this diagram it’s important to note that:
- the goal is to connect the Tier-0 Service Router (SR) to physical
- the Tier-0 Service Router connects to N-VDS External Segments shown as H2 and HS3
- the Edge VM NICs vNIC3 and vNIC4 connect to N-VDS External Segments Ext 1 and Ext 2
- Tier-0 Service Router traffic passes through two N-VDS segments to reach physical
- the TEP interface traffic passes through an Uplink Profile
Some typical causes of Edge connectivity issues:
- VLAN Tagging
- Edge Appliance Virtual NIC Connectivity
- MTU Setup
- Physical Setup
The first of these two root causes of connectivity issues will be covered in this post.
Issue 1: VLAN Tagging
Typically the physical uplink to the ESXi host is a trunked interface, where Edge traffic is expected to be tagged with a specific VLAN Tag ID, a number between 1 and 4094.
Considerations for VLAN ID numbering:
- Under IEEE 802.1Q, the maximum number of VLANs on a given Ethernet network is 4,094 (4,096 values provided by the 12-bit VID field minus reserved values at each end of the range, 0 and 4,095)
- VLAN ID 0 is reserved to mean untagged
- VLAN ID 4,095 is reserved for “Virtual Guest Tagging” (VGT).
- VLAN ID 0 – 4094 refers to an all VLANs permitted trunked interface
Locations to configure a VLAN Tag:
VLAN Tagging can be configured on segments, port groups, and uplink profiles, and can be applied to these three locations:
Location 1, Tier-0 SR External Segment:
Location 2, Edge VM NICs:
The Edge VM Virtual NICs can connect to VSS/VDS Port Groups or an N-VDS External Segments.
Location 3, Edge Uplink Profile:
Keep in mind that this uplink profile references “Transport VLAN” and is only applicable for Transport Overlay traffic to Compute Transport nodes and not to VLAN Overlay traffic. Keep this setting in mind when you’re working on establishing connectivity to Guest Workloads. This setting has no impact on Edge connectivity to physical.
Possible Tagging Scenarios
It’s important to remember that from Tier-0 SR to physical, traffic passes through two possible tagging locations.
Scenario 1: The physical network is not expecting Tier-0 SR traffic to be tagged, and with no tagging at Location 1 and 2, Tier-0 connects to physical.
Scenario 2: The physical network requires Tier-0 traffic to be tagged, which it is, at Location 2. Tier-0 connects to physical.
Scenario 3: The physical network requires Tier-0 traffic to be tagged, which it is at Location 1. To retain this tag, traffic is trunked at Location 2 with a range that includes the VLAN ID applied at Location 1. Tier-0 connects to physical.
Scenario 4: The physical network requires Tier-0 traffic to be tagged, which it is, at Location 1, but then again at Location 2. This double tagging results in traffic drops since it’s not possible to 802.1Q encapsulate a frame twice. (This scenario can occur in a subtle manner, for single N-VDS deployments, where Tier-0 and the Edge VM are configured to connect to the same external segment, and that segment is tagged. Since end to end traffic passes through the same Segment twice, traffic is double tagged, and Tier-0 connectivity to physical fails.)
Issue 2: Edge Appliance Virtual NIC Connectivity
Another setup issue that can occur with NSX-T Edges is the mapping of Edge fastpath interfaces to the correct Segment or Port Group.
Since there are four Edge interfaces, three of them fastpath, it is possible that the fastpath interfaces don’t get assigned to the appropriate N-VDS segment or VSS/VDS Port Group. This is more likely to occur if the Edge was deployed from an OVA, because the UI deployment method presents the fastpath mappings more clearly during the Edge deployment.
The NSX-T Data Center Installation Guide illustrates the following possible Link Setup for NSX Edge VM Networking
(It’s also possible to allocate fp-eth2 as a second uplink to a redundant TOR switch. )
In this scenario, to get Edge connectivity to physical established, it is essential to get fp-eth1 connected to the appropriate N-VDS segment or VSS/VDS Port Group. The reliable method to achieve this is by matching a NIC MAC address in NSX-T Manager UI with the Edge VM connections as seen using the vSphere Web Client.
From the NSX-T Web UI, edit the Edge N-VDS setup:
Notice that is fp-eth1 is the virtual NIC for the VLAN Transport Zone:
From the virtual NIC pull-down, note the MAC address associated with fp-eth1 is 00:50:56:96:6f:58:
From the Edge VM properties, via the vSphere Web Client, use MAC address 00:50:56:96:6f:58 to confirm that fp-eth1 is connected to the the appropriate N-VDS segment or VSS/VDS Port Group:
I hope this helps in demystifying some of the issues associated with Edge connectivity to the physical environment. There will be more to follow in future posts.
