[fusion_builder_container hundred_percent=”no” equal_height_columns=”no” menu_anchor=”” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” class=”” id=”” background_color=”” background_image=”” background_position=”center center” background_repeat=”no-repeat” fade=”no” background_parallax=”none” parallax_speed=”0.3″ video_mp4=”” video_webm=”” video_ogv=”” video_url=”” video_aspect_ratio=”16:9″ video_loop=”yes” video_mute=”yes” overlay_color=”” video_preview_image=”” border_size=”” border_color=”” border_style=”solid” padding_top=”” padding_bottom=”” padding_left=”” padding_right=””][fusion_builder_row][fusion_builder_column type=”1_1″ layout=”1_1″ background_position=”left top” background_color=”” border_size=”” border_color=”” border_style=”solid” border_position=”all” spacing=”yes” background_image=”” background_repeat=”no-repeat” padding_top=”” padding_right=”” padding_bottom=”” padding_left=”” margin_top=”0px” margin_bottom=”0px” class=”” id=”” animation_type=”” animation_speed=”0.3″ animation_direction=”left” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” center_content=”no” last=”no” min_height=”” hover_type=”none” link=””][fusion_text]
Introduction:
This post covers the minimum BGP configuration required, through the NSX-T 2.4 Simplified UI, to establish BGP peering and route propagation to the physical environment. This is the typical consideration once edge communication to physical has been established. This scenario consists of a two node Edge Cluster and a single external router.
There are a few things to note from this diagram:
- It would be nice to have two physical peers for additional diversity, but consider this a limitation of my lab environment.
- BGP peer IP addresses are shown in red.
- The BGP peers on the 192.168.100.0/24 segment were manually configured.
- The 169.254.0.130/25 IP is part of the auto-plumbed configuration when the Tier-1 gateway is deployed, and is a BGP peer to the Tier-0 Service Router, tier0_sr.
- The 169.254.0.130/25 peer allows BGP routes to propagate between Tier-0 and tier-1.
- The 192.168.70.0.24 subnet needs to propagate to physical.
- The 0.0.0.0/0 default route needs to propagate to ESXi hosts in the Cluster.
- The other subnets shown in black are also part of the auto-plumbed configuration, but are not part of the BGP architecture.
Objective:
This exercise will illustrate the base BGP configuration required to establish communication between virtualized workloads and the physical environment. This will cover the required configuration on both the Tier-0 and Tier-1 gateways and for a sample Cisco IOS based BGP peer.
Tier-0 Gateway Setup:
These next few images capture the Tier-0 configuration changes, where the bulk of the setup is performed.
1. Two Tier-0 gateway uplink interfaces are defined, one on edge1 and one on edge2:
2. A local BGP AS is defined on the main gateway configuration page. BGP is enabled by default:
3. Set Route Redistribution Advertised Tier-1 Subnet to Connected Subnets, where the goal is to announce the 192.168.70.0/24 route to physical:
4. Define the eBGP neighbor to physical:
Tier-1 Gateway Setup:
5. There is a single change required on the Tier-1 gateway, to Advertise all Connected Segments & Service ports, where once again the goal is to announce the 192.168.70.0/24 route to physical:
This wraps up the Simplified UI minimal changes required to enable BGP routing.
“Physical” Router configuration, Cisco 1000v:
In this lab environment, the role of the physical router is being performed by a Cisco Nexus 1000v virtual switch. Part of the configuration is shown here for reference: interface GigabitEthernet6 description BGP Peering Interface to NSX-T environment mtu 1600 ip address 192.168.100.2 255.255.255.0 ip nat inside negotiation auto cdp enable router bgp 65100 bgp log-neighbor-changes network 172.17.0.0 neighbor 192.168.100.102 remote-as 65111 neighbor 192.168.100.102 default-originate neighbor 192.168.100.103 remote-as 65111 neighbor 192.168.100.103 default-originate
NSX-T BGP configuration Useful Routing Related CLI Commands:
- Cisco Physical
- Router#sh version
- Router#sh ip interface brief
- Router#sh ip route
- Router#sh ip route bgp
- Router#sh ip bgp
- Router#sh ip bgp summary
- Router#show ip bgp neighbors <neighbor-ip> advertised-routes
- NSX-T Edge
- nsxtedge01> get maintenance-mode
- nsxtedge01> set maintenance-mode enable
- nsxtedge01> set maintenance-mode disable
- nsxtedge01(tier0_sr)> get int | find Name|IP|MAC|VNI|Interface
- nsxtedge01> get logical-routers
- vrf <vrf-id>
- nsxtedge01(tier0_sr)> get bgp
- nsxtedge01(tier0_sr)> get bgp neighbor summary
- nsxtedge01(tier0_sr)> get bgp neighbor <neighbor-ip> advertised-routes
nsxtedge01(tier0_sr)> get route - nsxtedge01(tier0_sr)> get forwarding
- nsxtedge01(tier1_sr)> get high-availability status
- ESXi Host CLI
- NSX-V legacy commands
- # net-vdr -I -l
- # net-vdr -R -l <router-id>
- NSX-V legacy commands
- ESXi Host nsxcli
- Commands introduced in NSX-T
- # nsxcli
- > get logical-router
- > get logical-router <router-id> interface
- > get logical-router <router-id> neighbor
- > get logical-router <router-id> forwarding
- Commands introduced in NSX-T
A Possible issue with this Minimal BGP Configuration:
Let's take a look at routes being advertised by edge2: edge02(tier0_sr)> get bgp neighbor 192.168.100.2 advertised-routes BGP table version is 25, local router ID is 192.168.100.103 Status flags: > - best, I - internal Origin flags: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path > 192.168.150.0/24 192.168.100.2 0 100 0 65100 ? > 10.160.110.0/24 192.168.100.2 0 100 0 65100 ? > 192.168.70.0/24 0.0.0.0 0 100 32768 ? > 169.254.0.128/25 0.0.0.0 0 100 32768 ? > 192.168.21.0/24 192.168.100.2 0 100 0 65100 ? > 10.155.14.0/24 192.168.100.2 0 100 0 65100 ? > 0.0.0.0/0 192.168.100.2 0 100 0 65100 i Here are the routes learned by the physical router: Router_DC01#show ip route bgp ..... Gateway of last resort is 192.168.21.1 to network 0.0.0.0 169.254.0.0/25 is subnetted, 1 subnets B 169.254.0.128 [20/0] via 192.168.100.103, 00:02:50 <--- unexpected B 192.168.70.0/24 [20/0] via 192.168.100.102, 00:02:50 <--- expected
There is a possible issue with this setup. Notice that the physical network has learned the required 192.168.70.0/24 route, but has also learned the 169.254.0.128/25 Intra Tier-1 route from edge2. It is possible that the BGP Domain administrator of the physical network may want the Intra Tier-1 route suppressed.
Possible Solutions to suppress the Intra Tier-1 route :
Possible solutions include:
- An IP Prefix list to only permit the 192.168.70.0/24 subnet outbound, applied to the eBGP neighbor definition.
- Announce a BGP super-net summary only route, such as 192.168.0.0/16, that covers 192.168.70.0/24, but not 169.254.0.128/25.
- A Route Map to only permit the 192.168.70.0/24 subnet outbound.
- Use acceptable auto-plumbed subnets.
- The physical BGP peer suppresses the 169.254.0.128/25 route inbound.
Sample Solution, using an IP Prefix List applied to the eBGP neighbor definition:
These next few images illustrate defining and applying an IP Prefix List to the eBGP neighbor:
Routing after the Prefix List has been applied:
vRouter_DC01#sh ip route bgp
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override
Gateway of last resort is 192.168.21.1 to network 0.0.0.0
B 192.168.70.0/24 [20/0] via 192.168.100.102, 23:08:49
This looks much better, with route suppression in place, just the 192.168.70.0/24 subnet is now propagating to physical!
------
As a result, here the is forwarding table from an ESXi host in the Compute Cluster:
[root@esxcna03-s1:~] nsxcli
esxcna03-s1.core.hypervizor.com> get logical-router
Logical Routers Summary
VDR UUID LIF num Route num f6826f40-7fe1-4882-92c7-f15b5ffd1a24 2 3 esxcna03-s1.core.hypervizor.com> get logical-router f6826f40-7fe1-4882-92c7-f15b5ffd1a24 forwarding Logical Routers Forwarding Table Flags Legend: [U: Up], [G: Gateway], [C: Connected], [I: Interface] [H: Host], [R: Reject], [B: Blackhole], [F: Soft Flush], [E: ECMP] Network Gateway Type Interface UUID 0.0.0.0/0 169.254.0.2 UG 456aff90-7b17-4599-8a83-324266e6f368 169.254.0.0/28 0.0.0.0 UCI 456aff90-7b17-4599-8a83-324266e6f368 192.168.70.0/24 0.0.0.0 UCI fae133d1-f989-4432-bbc4-fa94788aa7dd ::/0 fe80::50:56ff:fe56:5300 UG 456aff90-7b17-4599-8a83-324266e6f368 fe80::50:56ff:fe56:5300/128 :: UCI 456aff90-7b17-4599-8a83-324266e6f368 fe80:918:100:0:50:56ff:fe56:4452/128 :: UCI 456aff90-7b17-4599-8a83-324266e6f368 ff02:918:100::1:ff56:4452/128 :: UCI 456aff90-7b17-4599-8a83-324266e6f368 ff02:918:100::1:ff56:5300/128 :: UCI 456aff90-7b17-4599-8a83-324266e6f368 The desired default route, 0.0.0.0/0, resides in the Compute Cluster node.
A final note on Auto-plumbed Network IP Addressing:
It is possible to an create alternate auto-plumbed IP assignments, but there are some caveats with regards to the the IP range:
- It can only be specified in the Management Plane API during Tier-0 creation, not in the Advanced Networking and Security UI
- It is not currently available in Policy API, but this is expected in NSX-T 2.5, so it’s also not currently available in Simplified UI.
- If you already have IPs allocated from the default range, you cannot change the range.
You can find additional details on NSX-T BGP configuration on a Tier-0 Logical Router in the NSX-T Data Center Administration Guide.
Excellent write up, Gary! Thanks for this.
Mike, thanks so much for your continued support and guidance!