With the release of VMware NSX-T 3.0, there is now support for Federation in NSX-T
- You are able to scale NSX-T deployments by having NSX-T Manager cluster at each location, NSX-T manager at each location can possibly integrate with multiple vcenters present locally.
- Each location has Local NSX-T Manager with vcenters, clusters and hosts.
- NSX T Global Manager provides single pane of glass for management of NSX-T managers placed in the individual locations.
- Configuration change is done on the Active Global Manager which is then synced to Standby Global Manager.
Fabric preparation which includes creation of transport zones, host transport nodes, edge transport nodes is done using Local Manager.
- Tier 0 Gateways and Tier 1 Gateways can be stretched across multiple locations and here the number of locations are not just limited to two locations, you could also have a third location.
Segments can be stretched across multiple locations
- Ability to have non stretched Tier 1 Gateways in individual locations.
Here span of the segment could be limited to one location.
- Span of Tier 1 Gateway has to be less than or equal to the span of Tier 0 Gateway.
- Span of stretched Tier 1 Gateway which is DR only has to be equal to the span of stretched Tier 0 Gateway.
- Stretched Tier 1 Gateway can run services like edge firewall or NAT if required.
- NSX T Edge Clusters at each site are configured with RTEP IPs.
RTEPs are required to handle cross site traffic.
RTEPs are required only if the Edge cluster is used to configure a gateway that spans more than one location
RTEPs across sites in your setup should have IP connectivity to each other.
There is Geneve tunnel established between RTEPs.
Intra-location tunnel endpoints (TEP) and inter-location tunnel endpoints (RTEP) must use separate VLANs and layer 3 subnets.
This point is documented in admin guide of NSX-T version 3.0
|Vcenters, Clusters, Hosts and Edge Nodes|
In the lab setup:
1. There is vcenter in each location
2. NSX-T manager is also deployed in each location
3. There are two locations here - Bangalore and Delhi
4. NSX T Edge Clusters comprising of two edges are deployed in each location.
Two edges in Bangalore and two edges in Delhi.
4. Hosts in each location are prepared as Host Transport Nodes.
5. In my lab setup, there is only one Active Global Manager deployed.
In a production setup there will Standby Global Manager as well in other location.
6. Tier 0 Gateway has Active-Active availability mode.
7. Bangalore is Primary location and Delhi is configured as Secondary location with respect to NSX-T Federation.
Deployment of Global Manager is similar to deployment of Local Manager, you need to specify Location name and very importantly the role Global Manager while deploying Global NSX-T Manager.
Once, Global Manager is deployed then you are able to add locations
While adding a location, you need to specify
- Location Name
- SHA 256 thumbprint which is retrieved from Local Manager GUI.
|Adding location on Global Manager|
|Locations on Global Manager|
Fabric setup in Bangalore:
In each location, you need to prepare NSX-T Fabric which includes
- Configuring Transport Zones in each location
For now, the default nsx-overlay-transportzone works for Overlay traffic in each location.
- Preparing Compute Transport Nodes
- Configuring Edge Transport Nodes
- Configuring Edge Cluster
- Configure RTEPs on edge cluster.
Sharing the screenshots for these steps from Bangalore Location.
Please note that same steps are to be followed in the other location Delhi as part of fabric preparation.
IP Address Pools for Compute TEPs, Edge TEPs and RTEPs in Bangalore:
|Compute TEP Pool in Bangalore|
|Edge TEP Pool in Bangalore|
|RTEP Pool in Bangalore|
|Uplink Profiles for Host TNs and Edge TNs|
Above screenshot shows uplink profiles being created for edges and hosts in Bangalore location.
Since the edge is up linked to VDS (used for configuring NSX), separate VLANs are used for edge TEP and compute TEP respectively.
VLAN for compute TEP is 7
VLAN for edge TEP is 9
|Edge Transport Node Configuration for edge in Bangalore|
|Compute TN Configuration for host in Bangalore|
|Edge Cluster in Bangalore|
|Configure RTEP on edge cluster of Bangalore|
Configure RTEPs on edge cluster of Bangalore for cross site traffic.
Before proceeding to configurations on Global Manager, ensure that same steps are followed for preparing fabric in Delhi location.
RTEPs are to be created on the edge cluster in Delhi too.
There is RTEP to RTEP communication for cross site traffic.
Configurations on Global Manager:
With the fabric set up properly in both the locations - Bangalore and Delhi, now configuration can be applied on Global Manager.
VLAN 5 and VLAN 51 are used for uplink connectivity from edge to the physical routers in Bangalore.
VLANs 55 and 56 are used for uplink connectivity from edge to the physical routers in Delhi.
|Segment for uplink on Tier 0 Gateway using VLAN 5 in Bangalore|
|Segment for uplink on Tier 0 Gateway using VLAN 51 in Bangalore|
|Segment for uplink on Tier 0 Gateway using VLAN 55 in Delhi|
|Segment for uplink on Tier 0 using VLAN 56 in Delhi|
Above segments are created on Global Manager.
These segments are to be used while creating layer 3 interfaces on the stretched Tier 0 Gateway from Global Manager.
Next create stretched Tier 0 Gateway from Global Manager
|Create Tier 0 Gateway|
And Delhi location is set as secondary location for federation.
This Tier 0 Gateway has Active-Active availability mode as there is no requirement of services like edge firewall / NAT on this Tier 0 Gateway.
Active-Active availability mode provides greater throughput for North-South traffic.
|Layer 3 interfaces on Stretched Tier 0 Gateway|
Create Layer 3 interfaces on the stretched Tier 0 Gateway as shown above.
Here B1-V1-E1 corresponds to location Bangalore, VLAN 5 which is first uplink VLAN there, edge node 1 in Bangalore.
D1-V2-E4 corresponds to Delhi, VLAN 56 which is second uplink VLAN there, edge node 4 in Delhi.
And similar logic is used to create rest of the layer 3 interfaces on Tier 0 Gateway.
At this stage, we can test reach-ability between physical routers and Tier 0 Gateway interfaces.
Stretched Tier 0 Gateway uses BGP AS 65000
In the BGP configs, BGP AS number is configured and BGP neighbors are specified.
Enable route redistribution for connected Tier 0/Tier 1 subnets on stretched Tier 0 Gateway for both the locations - Bangalore and Delhi.
Physical routers in Delhi are prepending AS while receiving routes from edges in Delhi.
|Create stretched Tier 1 Gateway from Global Manager|
In the lab setup, I have not used custom span so as to require edge cluster association with T1.
This Tier 1 Gateway is DR only Tier 1.
Span of this Tier 1 Gateway is the same as the span of Tier 0 Gateway spanning both locations Bangalore and Delhi.
Advertise connected subnets on Tier 1 Gateway.
|Create stretched segment from Global Manager|
Traffic type is Overlay
Now attach workloads to this segment and test East-West Connectivity and North-South connectivity.
|Ping from VM in Bangalore to VM in Delhi|
|Ping from VM in Delhi to VM in Bangalore|
|Verify RTEP to RTEP communication|
|Routing tables of Tier 1 DR and Tier 0 DR on ESXi 12 in Delhi|
VM in Delhi is on ESXi 12
Above shows the routing tables corresponding to Tier 1 DR and Tier 0 DR of ESXi 12
|Trace from VM in Delhi to loopback on physical router 2 of Delhi|
|Traceflow from VM in Delhi location to loopback of physical router in Delhi|
|RTEP interface on Edge Node 2 in Bangalore location|
From the above output, we see that traffic lands on Edge Node 4 in Delhi, then goes through the tunnel between RTEPs and lands on Edge Node 2 in Bangalore location. Edge Node 2 then routes it towards the physical network.