NSX-T East-West Traffic Flow (Part 1)

Introduction:

Traffic flow can be defined in the following directions: North-South and East-West.

• North-South traffic is traffic leaving or entering the NSX-T domain, for example, a virtual machine on an overlay network communicating with a device on the physical network.
• East-West traffic is traffic that remains in the NSX-T domain, for example, two virtual machines on the same or different logical switches communicating with each other.

NSX-T East-West Traffic Flow is Part 1 of a two part series, taking a close close at traffic flow in an NSX-T environment.

Lab setup for testing a number of East-West traffic flow scenarios:

Ronald de Jong’s excellent article on Distributed Multi-Tier Routing in NSX-T had me thinking about the NSX-T Edges’s role in East-West traffic flow. I was also interested in the distributed capability of an NSX-T Tier-1 router. To get a better understanding, I decided to run through a number of East-West traffic flow lab scenarios under the following lab setup:

Notes on the Lab Topology:

The Tier-0 Gateway must be instantiated on an Edge Cluster:

When adding a Tier-0 Gateway to the environment, you must specify an Edge Clutser on which the gateway is instantiated.

The Tier-1 Gateways can be optionally instantiated on an Edge Cluster:

When adding a Tier-1 Gateway to the environment, specifying the Edge Cluster is optional. The Edge Cluster is specified if you plan to configure stateful services such as NAT on the Tier-1 Gateway. In my lab, it’s important to note that the Edge Cluster is not currently specified. This setting will have an impact on East-West flow in some scenarios.

Since the Tier-1 Gateways are not instantiated on the Edge Cluster, we are expecting Tier-1 Distributed Routers (DR), but not Tier-1 Service Routers (SR):

nsxtedge02> get logical-routers
 Logical Router
 UUID                                   VRF    LR-ID  Name                              Type                        Ports
 736a80e3-23f6-5a2d-81d6-bbefb2786666   0      0                                        TUNNEL                      3
 a3a92312-a01e-45c4-b9f8-4b1520b4a73f   2      8193   SR-lab-tier-0                     SERVICE_ROUTER_TIER0        6
 3ef116ea-7adc-48bb-bc89-89fd16502087   3      6146   DR-lab-tier-0                     DISTRIBUTED_ROUTER_TIER0    5
 c1763624-cfe9-44d2-96e3-c2413107a22e   5      11266  DR-lab-tier-1-tenant-2            DISTRIBUTED_ROUTER_TIER1    5
 9d278256-3211-425f-afbe-0011be89876b   6      12289  DR-lab-tier-1-tenant-1            DISTRIBUTED_ROUTER_TIER1    6

From this we can see that there are:

• A Tier-0 SR and a DR, since these were instantiated in the Edge Cluster
• Tier-1 DRs only for the Tennant Gateways, since these were not instantiated in the Edge Cluster

Test Scenarios:

This lab topology, with carefully positioned Guest VMs, provides the ability to follow East-West traffic through NSX-T components. I came up with six scenarios to test:

Scenario 1:

• Guest VMs on same ESXi host, same segment

The NSX-T Traceflow utility is an excellent method to visualize the flow between these two Guest VMs. Notice here that VM1 and VM2 are selected within the utility:

Here is the resulting Traceflow, between two Guest VMs, on the same host and the same Segment, with the Distributed Firewall enabled at the transport node level. In this scenario, traffic essentially passes through DFW rules:

I then disable DFW at the transport node level, following this article, and then run the Traceflow again with the same two endpoints. Notice that Distributed Firewall no longer appears as a Component in the flow:

For now, I will keep DFW disabled at the transport node level to simplify the Traceflow output for traffic flow analysis.

Scenario 2:

• Guest VMs on different ESXi hosts, same segment:

Notice here that traffic passes between ESXi hosts over a Geneve tunnel.

Scenario 3:

• Guest VMs on same ESXi host, different segments:

Notice here that traffic remains on the single ESXi host.

Scenario 4:

• Guest VMs on different ESXi hosts, different segments:

Notice here that once again traffic passes between ESXi hosts over a Geneve tunnel.

Scenario 5:

• Guest VMs on Different Tier-1 routers, same ESXi host, different segments:

Notice that this traffic never leaves Transport Node ESXCNA01:

Scenario 6:

• Guest VMs on Different Tier-1 routers, different ESXi hosts, different segments:

Once again, as in all these scenarios, transport Nodes were only ESXi hosts, and never Edge Cluster nodes, in spite of traffic traversing the Tier-0 gateway.

Summary of Results:

When the Tier-1 Gateways are not instantiated on an Edge Cluster:

• The Traceflow hop Transport Nodes were only ESXi hosts, and never Edge Cluster nodes.
• This demonstrates the Distributed capability of NSX-T; the ability to move traffic between Tiers, Hosts, and Segments without it passing through the Edge Cluster.

That’s it for now. In Part 2 of this series, we will examine the implications of instantiating the Tier-1 Gateway on the Edge Cluster.