Introduction:
NSX-T East-West Traffic Flow is Part 2 in a 2 part series, continuing the close look at traffic flow in an NSX-T environment. In Part 1 of this series, we looked at a set of scenarios where Tier-1 Gateways were not instantiated on the Edge Cluster. In this post we will reexamine traffic flows where Tier-1 Gateways are instantiated on the Edge Cluster
Lab setup for testing a number of East-West traffic flow scenarios:
As a reminder, this is the lab topology that we will continue to use.
Notes on the Lab Topology:
Remember that the Tier-0 Gateway must be instantiated on an Edge Cluster:
When adding a Tier-0 Gateway to the environment, you must specify an Edge Clutser on which the gateway is instantiated.
The Tier-1 Gateways can be optionally instantiated on an Edge Cluster:
When adding a Tier-1 Gateway to the environment, specifying the Edge Cluster is optional. The Edge Cluster is specified if you plan to configure stateful services such as NAT on the Tier-1 Gateway. In my lab, it’s important to note that in Part 1, the Edge Cluster was not specified. This setting has an impact on East-West flow in some scenarios.
Since the Tier-1 Gateways are not instantiated on the Edge Cluster, we are expecting Tier-1 Distributed Routers (DR), but not Tier-1 Service Routers (SR):
nsxtedge02> get logical-routers Logical Router UUID VRF LR-ID Name Type Ports 736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 3 a3a92312-a01e-45c4-b9f8-4b1520b4a73f 2 8193 SR-lab-tier-0 SERVICE_ROUTER_TIER0 6 3ef116ea-7adc-48bb-bc89-89fd16502087 3 6146 DR-lab-tier-0 DISTRIBUTED_ROUTER_TIER0 5 c1763624-cfe9-44d2-96e3-c2413107a22e 5 11266 DR-lab-tier-1-tenant-2 DISTRIBUTED_ROUTER_TIER1 5 9d278256-3211-425f-afbe-0011be89876b 6 12289 DR-lab-tier-1-tenant-1 DISTRIBUTED_ROUTER_TIER1 6
From this we can see that there is:
- A Tier-0 SR and a DR, since these were instantiated in the Edge Cluster
- Tier-1 DRs only for the Tennant Gateways, since these were not instantiated in the Edge Cluster
Instantiating Tier-1 Gateways on the Edge Cluster
Let’s instantiate both tenant Tier-1 routers on the Edge Cluster – creatively named “Edge Cluster”:
This same change has been applied to lab-tier-1-tennant-1 and lab-tier-1-tennant-2.
Since the Tier-1 Gateways are instantiated on the Edge Cluster, we are expecting Tier-1 Distributed Routers (DR), and now the Tier-1 Service Routers (SR):
nsxtedge02> get logical-routers Logical Router UUID VRF LR-ID Name Type Ports 736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 3 3ef116ea-7adc-48bb-bc89-89fd16502087 1 6146 DR-lab-tier-0 DISTRIBUTED_ROUTER_TIER0 5 9d278256-3211-425f-afbe-0011be89876b 2 12289 DR-lab-tier-1-tenant-1 DISTRIBUTED_ROUTER_TIER1 5 c1763624-cfe9-44d2-96e3-c2413107a22e 3 11266 DR-lab-tier-1-tenant-2 DISTRIBUTED_ROUTER_TIER1 4 a3a92312-a01e-45c4-b9f8-4b1520b4a73f 4 8193 SR-lab-tier-0 SERVICE_ROUTER_TIER0 6 2938a6d8-c129-4f7e-8356-ce696d07738e 5 13313 SR-lab-tier-1-tenant-1 SERVICE_ROUTER_TIER1 5 34823c67-1efd-49b6-b495-29dec792f377 6 14337 SR-lab-tier-1-tenant-2 SERVICE_ROUTER_TIER1 5
Test Scenarios:
Let’s now run through the same six scenarios that we did in Part 1:
Scenario 1:
- Guest VMs on same ESXi host, same segment
The NSX-T Traceflow utility is an excellent method to visualize the flow between these two Guest VMs. Notice here that VM1 and VM2 are selected within the utility:
Here is the resulting Traceflow between two Guest VMs on the same host and the same Segment with the Distributed Firewall disabled at the transport node level.
This is the same result as from NSX-T East-West Traffic Flow (Part 1)
Scenario 2:
- Guest VMs on different ESXi hosts, same segment:
Notice here that traffic passes between ESXi hosts over a Geneve tunnel.
This is the same result as from NSX-T East-West Traffic Flow (Part 1)
Scenario 3:
- Guest VMs on same ESXi host, different segments:
This is the same result as from NSX-T East-West Traffic Flow (Part 1)
Scenario 4:
- Guest VMs on different ESXi hosts, different segments:
Notice here that once again traffic passes between ESXi hosts over a Geneve tunnel.
This is the same result as from NSX-T East-West Traffic Flow (Part 1)
Scenario 5:
- Guest VMs on Different Tier-1 routers, same ESXi host, different segments:
Notice that this is the first time in all of these test scenarios that traffic needed to extend beyond ESXi hosts ESXCNA01 and ESXCNA02. Notice in this scenario that traffic does traverse Transport Node Edge nsxtedge02, resulting in a significantly longer path:
Scenario 6:
- Guest VMs on Different Tier-1 routers, different ESXi hosts, different segments:
Once again, notice in this scenario that traffic does traverse Transport Node Edge nsxtedge02, resulting in a significantly longer path:
Summary of Results:
The following table summarizes results from Part 1 and Part 2 of this series:
- Tier-0 gateways must be instantiated on an Edge Cluster.
- Tier-1 gateways may be instantiated on an Edge Cluster.
- When adding a Tier-1 Gateway to the environment, specifying the Edge Cluster is required if you plan to configure stateful services such as NAT or fire-walling on the Tier-1 Gateway.
- In most scenarios traffic flow is the same whether Tier-1 gateways are instantiated on a Edge Cluster or not.
- Tier-1 instantiation on an Edge Cluster has implications in terms of inter-tenant traffic flow, traffic between Tier-1 gateways.
- When the Tier-1 Gateways are not instantiated on an Edge Cluster, inter-tenant Tier-1 traffic does not traverse the Edge. Traffic remains on ESXi hosts only.
- When the Tier-1 Gateways are instantiated on an Edge Cluster, inter-tenant Tier-1 traffic is expected to traverse the Edge, resulting in a significantly longer path
