Tuesday, April 21, 2020

NSX-T Federation (Local Egress Using Tier 0 Active-Active, All Locations Primary)

NSX-T Federation

Local Egress Using Tier 0 Active-Active, All Locations Primary

This is a follow up to my earlier post on NSX-T Federation.

This design uses Local Egress similar to the Local Egress feature in NSX-V.

One has to note that if there are firewalls upstream to Tier 0 Gateway in both the locations, then firewall in one site can drop the connection just because the return packet is actually destined to the other firewall.
This setup uses Asymmetric routing.
Traffic from NSX can egress out via physical routers in Delhi and the return traffic can ingress towards NSX setup using physical routers in Bangalore.

In the lab setup, there is:
 
a. Stretched Tier 0 Gateway across both the locations Bangalore and Delhi
b. Stretched Tier 1 Gateway across both the locations.
c. Tier 1 Gateway is a DR only Tier 1 Gateway.
d. Segment attached to Tier 1 Gateway is a stretched segment across both the locations.
e. Both the locations are Primary here.
f. AS Path prepending configuration is not applied on the physical routers or on the Tier 0 Gateway.

Lab Topology Diagrams: 

Logical Topology for the Lab Setup

BGP AS Setup

Configuration:

You may follow the steps outlined in the earlier post for:

- Preparing NSX-T Fabric.
- Creating stretched Tier 0 Gateway
- Creating Layer 3 interfaces on this stretched Tier 0 Gateway.
- Applying BGP configuration on stretched Tier 0 Gateway.
- Creating a stretched DR only Tier 1 Gateway across both the locations.
- Creating stretched segment.

Let us jump into the validation part.

Tier 0 Gateway configured with all locations as Primary

Validation:

Local egress behavior is noticed for the traffic flows.

Ping and trace from loopback on physical router in Delhi towards VM 172.16.10.3 in Delhi

Ping and trace from loopback on physical router in Bangalore towards VM in Delhi location

Trace from VM in Delhi location to loopback IP of Delhi router goes egress out of edges in Delhi directly to Delhi router


Trace from VM in Delhi towards loopback network on physical router of Delhi

The above trace flow is from VM in Delhi location towards loopback network configured on physical router of Delhi location.

Traffic leaves the hypervisor in Delhi location and uses its TEP interface to encapsulate the original packet and send the traffic towards TEP interface on edge in Delhi location.

This edge de encapsulates the packet and forwards it towards the router locally available in Delhi.
 

Trace from VM in Delhi towards loopback network on physical router of Bangalore



Trace from VM in Bangalore towards loopback network on physical router of Delhi

Trace from VM in Bangalore towards loopback network on physical router of Bangalore

Wednesday, April 15, 2020

Federation in NSX-T 3.0

NSX-T Federation

With the release of VMware NSX-T 3.0, there is now support for Federation in NSX-T

With Federation:

- You are able to scale NSX-T deployments by having NSX-T Manager cluster at each location, NSX-T manager at each location can possibly integrate with multiple vcenters present locally.

- Each location has Local NSX-T Manager with vcenters, clusters and hosts.
 
- NSX T Global Manager provides single pane of glass for management of NSX-T managers placed in the individual locations.

- Active and Standby Global Managers are placed in two different locations.

- Configuration change is done on the Active Global Manager which is then synced to Standby Global Manager.
Fabric preparation which includes creation of transport zones, host transport nodes, edge transport nodes is done using Local Manager.

- Tier 0 Gateways and Tier 1 Gateways can be stretched across multiple locations and here the number of locations are not just limited to two locations, you could also have a third location.
Segments can be stretched across multiple locations

- Ability to have non stretched Tier 1 Gateways in individual locations.
Here span of the segment could be limited to one location.

- Span of Tier 1 Gateway has to be less than or equal to the span of Tier 0 Gateway.

- Span of stretched Tier 1 Gateway which is DR only has to be equal to the span of stretched Tier 0 Gateway.

- Stretched Tier 1 Gateway can run services like edge firewall or NAT if required.

- NSX T Edge Clusters at each site are configured with RTEP IPs.
RTEPs are required to handle cross site traffic.
RTEPs are required only if the Edge cluster is used to configure a gateway that spans more than one location
RTEPs across sites in your setup should have IP connectivity to each other.
There is Geneve tunnel established between RTEPs.

Intra-location tunnel endpoints (TEP) and inter-location tunnel endpoints (RTEP) must use separate VLANs and layer 3 subnets.
This point is documented in admin guide of NSX-T version 3.0

Lab Setup:

Vcenters, Clusters, Hosts and Edge Nodes


Lab Setup

In the lab setup:
1. There is vcenter in each location
2. NSX-T manager is also deployed in each location
3. There are two locations here - Bangalore and Delhi
4. NSX T Edge Clusters comprising of two edges are deployed in each location.
Two edges in Bangalore and two edges in Delhi.
4. Hosts in each location are prepared as Host Transport Nodes.
5. In my lab setup, there is only one Active Global Manager deployed.
In a production setup there will Standby Global Manager as well in other location.
6. Tier 0 Gateway has Active-Active availability mode.
7. Bangalore is Primary location and Delhi is configured as Secondary location with respect to NSX-T Federation.

Deployment of Global Manager is similar to deployment of Local Manager, you need to specify Location name and very importantly the role Global Manager while deploying Global NSX-T Manager.

Once, Global Manager is deployed then you are able to add locations
While adding a location, you need to specify
- Location Name
- FQDN
- Username
- Password
- SHA 256 thumbprint which is retrieved from Local Manager GUI.



Adding location on Global Manager



Locations on Global Manager
Above you see that two locations Bangalore and Delhi have been added to the Global Manager.

Fabric setup in Bangalore:

In each location, you need to prepare NSX-T Fabric which includes
- Configuring Transport Zones in each location
For now, the default nsx-overlay-transportzone works for Overlay traffic in each location.
- Preparing Compute Transport Nodes
- Configuring Edge Transport Nodes
- Configuring Edge Cluster
- Configure RTEPs on edge cluster.

Sharing the screenshots for these steps from Bangalore Location.
Please note that same steps are to be followed in the other location Delhi as part of fabric preparation.

IP Address Pools for Compute TEPs, Edge TEPs and RTEPs in Bangalore:


Compute TEP Pool in Bangalore


Edge TEP Pool in Bangalore


 RTEP Pool in Bangalore


Uplink Profiles for Host TNs and Edge TNs
Uplink profiles are used while configuring NSX on edges and the hosts.
Above screenshot shows uplink profiles being created for edges and hosts in Bangalore location.
Since the edge is up linked to VDS (used for configuring NSX), separate VLANs are used for edge TEP and compute TEP respectively.
VLAN for compute TEP is 7
VLAN for edge TEP is 9




Edge Transport Node Configuration for edge in Bangalore


Compute TN Configuration for host in Bangalore
Above screenshots show edge transport node configuration & compute transport node configuration respectively.



Edge Cluster in Bangalore
Above shows edge cluster in Bangalore


Configure RTEP on edge cluster of Bangalore

Configure RTEPs on edge cluster of Bangalore for cross site traffic.

Before proceeding to configurations on Global Manager, ensure that same steps are followed for preparing fabric in Delhi location.
RTEPs are to be created on the edge cluster in Delhi too.
There is RTEP to RTEP communication for cross site traffic. 

Configurations on Global Manager:

With the fabric set up properly in both the locations - Bangalore and Delhi, now configuration can be applied on Global Manager.


VLAN 5 and VLAN 51 are used for uplink connectivity from edge to the physical routers in Bangalore.
VLANs 55 and 56 are used for uplink connectivity from edge to the physical routers in Delhi.

Segment for uplink on Tier 0 Gateway using VLAN 5 in Bangalore



Segment for uplink on Tier 0 Gateway using VLAN 51 in Bangalore





Segment for uplink on Tier 0 Gateway using VLAN 55 in Delhi


Segment for uplink on Tier 0 using VLAN 56 in Delhi


Above segments are created on Global Manager.

These segments are to be used while creating layer 3 interfaces on the stretched Tier 0 Gateway from Global Manager.


Next create stretched Tier 0 Gateway from Global Manager


Create Tier 0 Gateway
While creating Tier 0 Gateway, Bangalore location is set as primary for federation.
And Delhi location is set as secondary location for federation.
This Tier 0 Gateway has Active-Active availability mode as there is no requirement of services like edge firewall / NAT on this Tier 0 Gateway.
Active-Active availability mode provides greater throughput for North-South traffic.


Layer 3 interfaces on Stretched Tier 0 Gateway

Create Layer 3 interfaces on the stretched Tier 0 Gateway as shown above.

Here B1-V1-E1 corresponds to location Bangalore, VLAN 5 which is first uplink VLAN there, edge node 1 in Bangalore.
D1-V2-E4 corresponds to Delhi, VLAN 56 which is second uplink VLAN there, edge node 4 in Delhi.

And similar logic is used to create rest of the layer 3 interfaces on Tier 0 Gateway.

At this stage, we can test reach-ability between physical routers and Tier 0 Gateway interfaces.




BGP Setup:


BGP Setup
Physical routers are in BGP AS 65001
Stretched Tier 0 Gateway uses BGP AS 65000







In the BGP configs, BGP AS number is configured and BGP neighbors are specified.


Enable route redistribution for connected Tier 0/Tier 1 subnets on stretched Tier 0  Gateway for both the locations - Bangalore and Delhi.

Physical routers in Delhi are prepending AS while receiving routes from edges in Delhi.



Create stretched Tier 1 Gateway from Global Manager
 
From the Global Manager, create a stretched Tier 1 Gateway.
In the lab setup, I have not used custom span so as to require edge cluster association with T1.
This Tier 1 Gateway is DR only Tier 1.
Span of this Tier 1 Gateway is the same as the span of Tier 0 Gateway spanning both locations Bangalore and Delhi.

Advertise connected subnets on Tier 1 Gateway.




Create stretched segment from Global Manager

While creating overlay segment, specify the Tier 1 Gateway and gateway IP.
Traffic type is Overlay

Now attach workloads to this segment and test East-West Connectivity and North-South connectivity.

Validation:


Ping from VM in Bangalore to VM in Delhi


Ping from VM in Delhi to VM in Bangalore
From the above, we verify the MAC addresses associated with the VMs and that there is reach-ability between the VMs in different locations.


Verify RTEP to RTEP communication
Above we see that RTEP to RTEP tunnel is established properly.


Traffic flow from loopback of physical router in Delhi to VM on the segment


The above trace shows the traffic from loopback of physical router in Delhi destined to VM on segment (this VM is on host ESXi 12 of location Delhi) goes through Bangalore.

Above output also shows that route through location Bangalore has shorter AS Path because AS Path prepending is applied on physical router of Delhi 
 
Routing tables of Tier 1 DR and Tier 0 DR on ESXi 12 in Delhi

VM in Delhi is on ESXi 12
Above shows the routing tables corresponding to Tier 1 DR and Tier 0 DR of ESXi 12


Trace from VM in Delhi to loopback on physical router 2 of Delhi


Trace from VM in Bangalore to loopback address on physical router 2 of Delhi
Traceflow from VM in Delhi location to loopback of physical router in Delhi

RTEP interface on Edge Node 2 in Bangalore location

From the above output, we see that traffic lands on Edge Node 4 in Delhi, then goes through the tunnel between RTEPs and lands on Edge Node 2 in Bangalore location. Edge Node 2 then routes it towards the physical network.