NSX-T Federation - Active Active Data Centers

 

 NSX-T Federation - Active Active Data Centers

 

This blog covers NSX-T Federation feature which allows L2 stretching between Data Centers as well as supports micro segmentation for workloads based on security tags.

Earlier blogs covered NSX-T Federation with a single Tier 0 stretched Gateway.

Here we explore how two Tier 0 Gateways can be utilized for workloads which are active in both data centers.

Logical Setup used in Lab

 

The above setup is used in the lab.

 

A brief about the setup:

1. Global Manager sits in Bangalore

2. Both sites have local manager

3. Hosts in each site have been prepared as hosts transport nodes.

4. Edges have been deployed in each site and configured as edge transport nodes.

5. Each site has site local uplink VLANs, edge TEP VLAN, host TEP VLAN & RTEP VLAN

RTEP interfaces are instantiated on edges to handle inter site traffic.

Every stretched segment will have local edges designated as Active and Standby for that specific segment.

6. Four edges correspond to Tier 0 Gateway Bangalore which has Bangalore as Primary location and Delhi as secondary location.

One edge cluster in Bangalore and another in Delhi

7. Four edges correspond to Tier 0 Gateway Delhi which has Delhi as Primary location and Bangalore as secondary location.

One edge cluster in Bangalore and another in Delhi

8. Transport zone configuration, edge node configuration and hosts transport node configuration is done from Local Manager.

9. Stretched Tier 0 Gateway, segments used for uplinks of stretched Tier 0 Gateway and stretched Tier 1 Gateway is created from Global Manager UI.

Segments connected to stretched Tier 1 Gateways are also created from Global Manager UI.

10. A total of eight edge nodes are configured in this lab setup.

IP Addressing and VLAN details

NSX-T Fabric for Bangalore Location

IP Address Pools for Compute TEP, Edge TEP and RTEPs

Local Transport Zones in Bangalore

Uplink Profiles in Bangalore

 

Compute Transport Node Profile

Edge Node Connectivity

In the above diagram, edge is connected to VDS which was earlier used to configure NSX on hosts.

VLAN backed trunk segments are created on hosts' VDS for uplink connectivity of edge.

Fast path interfaces of edge are connected to these trunk segments.

Edge Transport Node Configuration

 

 

Host Transport Nodes are configured

 

Once hosts are configured for NSX, tunnel endpoint interfaces are created on the hosts, NSX-T software is installed on the hosts.

 

 

Edge Transport Nodes are configured

 

Once edges are configured for NSX, tunnel endpoint interfaces are created on edges and the edge is connected to appropriate trunk segments on host VDS.

 

Edge Clusters in Bangalore

 

 

RTEP configuration on edge clusters of Bangalore

RTEP configuration is applied to both edge clusters in Bangalore

Before applying configurations on Global Manager, ensure that below configurations are also applied in other location:

a. Transport Zones

b. IP Pools

c. Uplink Profiles

d. Compute Transport Node Profiles

e. Edge Transport Nodes config

f. Hosts are configured as host transport nodes.

g. RTEP configs on edge clusters in Delhi

BGP Setup

 

BGP Setup for Tier 0 Gateway Bangalore

BGP AS 65000 is used on Tier 0 Gateway Bangalore

e BGP is used between Tier 0 Gateway and upstream routers.

Physical network is under AS 65001

Traffic ingress and egress to/from subnet connected to Tier 1 Gateway Bangalore goes through physical routers in Bangalore.

This gives deterministic traffic flow.

AS Path prepending is used on physical routers of Delhi Location to influence this traffic flow.

Physical routers are sending a default route on a per BGP peer basis.

Routes from NSX are redistributed into BGP

BGP Setup for Tier 0 Gateway Delhi

 

BGP AS 65002 is used on Tier 0 Gateway Bangalore

e BGP is used between Tier 0 Gateway and upstream routers.

Physical network is under AS 65001

Traffic ingress and egress to/from subnet connected to Tier 1 Gateway Delhi goes through physical routers in Delhi

This gives deterministic traffic flow.

AS Path prepending is used on physical routers of Bangalore Location to influence this traffic flow.

Physical routers are sending a default route on a per BGP peer basis.

Routes from NSX are redistributed into BGP

 

Global Manager Configuration

Locations are added to Global Manager

Segments are created on Global Manager for uplink connectivity of Tier 0 Gateway

While creating segments on Global Manager, specify location, local transport zone and the VLAN ID

Tier 0 Gateway Bangalore

 

While creating stretched Tier 0 Gateway, specify the edge cluster and the corresponding location as Primary or Secondary.

For Tier 0 Gateway Bangalore, primary location is Bangalore.

For Tier 0 Gateway Delhi, primary location is Delhi

L3 interfaces on Tier 0 Gateway Bangalore

Likewise Tier 0 Gateway is created with Delhi as primary location & Bangalore as secondary location.

Tier 1 Gateway config on Global Manager

Next create stretched Tier 1 Gateway and connect to the already defined Tier 0 Gateway.

Tier 1 Gateways on Global Manager

 

 

Segments on Global Manager connected to Tier 1 Gateway

Next deploy VMs and connect them to appropriate segments.

 

 

VM connected to Overlay Network

Validation

Trace from router in Delhi to VM behind Tier 1 Gateway Bangalore goes through physical router of Bangalore

Trace from router in Bangalore to VM behind Tier 1 Gateway Delhi goes through physical router of Delhi

 

Trace from loopback of second physical router in Delhi to VM behind Tier 1 Gateway Bangalore goes through physical router 1 of Bangalore location

Trace from VM behind Tier 1 Gateway Bangalore towards loopback of physical router 2 in Delhi goes through physical router 2 of Bangalore location

 

Trace from VM in Delhi to loopback of physical router 1 in Bangalore goes through physical router of Delhi

Trace from loopback of physical router 2 in Bangalore to VM behind Tier 1 Gateway Delhi goes through physical router in Delhi

RTEP to RTEP tunnel is established

NSX-T Federation (Local Egress Using Tier 0 Active-Active, All Locations Primary)

NSX-T Federation

Local Egress Using Tier 0 Active-Active, All Locations Primary

This is a follow up to my earlier post on NSX-T Federation.

This design uses Local Egress similar to the Local Egress feature in NSX-V.

One has to note that if there are firewalls upstream to Tier 0 Gateway in both the locations, then firewall in one site can drop the connection just because the return packet is actually destined to the other firewall.

This setup uses Asymmetric routing.

Traffic from NSX can egress out via physical routers in Delhi and the return traffic can ingress towards NSX setup using physical routers in Bangalore.

In the lab setup, there is:

 

a. Stretched Tier 0 Gateway across both the locations Bangalore and Delhi

b. Stretched Tier 1 Gateway across both the locations.

c. Tier 1 Gateway is a DR only Tier 1 Gateway.

d. Segment attached to Tier 1 Gateway is a stretched segment across both the locations.

e. Both the locations are Primary here.

f. AS Path prepending configuration is not applied on the physical routers or on the Tier 0 Gateway.

Lab Topology Diagrams: 

Logical Topology for the Lab Setup

BGP AS Setup

Configuration:

You may follow the steps outlined in the earlier post for:

- Preparing NSX-T Fabric.

- Creating stretched Tier 0 Gateway

- Creating Layer 3 interfaces on this stretched Tier 0 Gateway.

- Applying BGP configuration on stretched Tier 0 Gateway.

- Creating a stretched DR only Tier 1 Gateway across both the locations.

- Creating stretched segment.

Let us jump into the validation part.

Tier 0 Gateway configured with all locations as Primary

Validation:

Local egress behavior is noticed for the traffic flows.

Ping and trace from loopback on physical router in Delhi towards VM 172.16.10.3 in Delhi

Ping and trace from loopback on physical router in Bangalore towards VM in Delhi location

Trace from VM in Delhi location to loopback IP of Delhi router goes egress out of edges in Delhi directly to Delhi router

Trace from VM in Delhi towards loopback network on physical router of Delhi

The above trace flow is from VM in Delhi location towards loopback network configured on physical router of Delhi location.

Traffic leaves the hypervisor in Delhi location and uses its TEP interface to encapsulate the original packet and send the traffic towards TEP interface on edge in Delhi location.

This edge de encapsulates the packet and forwards it towards the router locally available in Delhi.
 

Trace from VM in Delhi towards loopback network on physical router of Bangalore

Trace from VM in Bangalore towards loopback network on physical router of Delhi

Trace from VM in Bangalore towards loopback network on physical router of Bangalore