NSX-T Federation - Active Active Data Centers

 

 NSX-T Federation - Active Active Data Centers

 

This blog covers NSX-T Federation feature which allows L2 stretching between Data Centers as well as supports micro segmentation for workloads based on security tags.

Earlier blogs covered NSX-T Federation with a single Tier 0 stretched Gateway.

Here we explore how two Tier 0 Gateways can be utilized for workloads which are active in both data centers.

Logical Setup used in Lab

 

The above setup is used in the lab.

 

A brief about the setup:

1. Global Manager sits in Bangalore

2. Both sites have local manager

3. Hosts in each site have been prepared as hosts transport nodes.

4. Edges have been deployed in each site and configured as edge transport nodes.

5. Each site has site local uplink VLANs, edge TEP VLAN, host TEP VLAN & RTEP VLAN

RTEP interfaces are instantiated on edges to handle inter site traffic.

Every stretched segment will have local edges designated as Active and Standby for that specific segment.

6. Four edges correspond to Tier 0 Gateway Bangalore which has Bangalore as Primary location and Delhi as secondary location.

One edge cluster in Bangalore and another in Delhi

7. Four edges correspond to Tier 0 Gateway Delhi which has Delhi as Primary location and Bangalore as secondary location.

One edge cluster in Bangalore and another in Delhi

8. Transport zone configuration, edge node configuration and hosts transport node configuration is done from Local Manager.

9. Stretched Tier 0 Gateway, segments used for uplinks of stretched Tier 0 Gateway and stretched Tier 1 Gateway is created from Global Manager UI.

Segments connected to stretched Tier 1 Gateways are also created from Global Manager UI.

10. A total of eight edge nodes are configured in this lab setup.

IP Addressing and VLAN details

NSX-T Fabric for Bangalore Location

IP Address Pools for Compute TEP, Edge TEP and RTEPs

Local Transport Zones in Bangalore

Uplink Profiles in Bangalore

 

Compute Transport Node Profile

Edge Node Connectivity

In the above diagram, edge is connected to VDS which was earlier used to configure NSX on hosts.

VLAN backed trunk segments are created on hosts' VDS for uplink connectivity of edge.

Fast path interfaces of edge are connected to these trunk segments.

Edge Transport Node Configuration

 

 

Host Transport Nodes are configured

 

Once hosts are configured for NSX, tunnel endpoint interfaces are created on the hosts, NSX-T software is installed on the hosts.

 

 

Edge Transport Nodes are configured

 

Once edges are configured for NSX, tunnel endpoint interfaces are created on edges and the edge is connected to appropriate trunk segments on host VDS.

 

Edge Clusters in Bangalore

 

 

RTEP configuration on edge clusters of Bangalore

RTEP configuration is applied to both edge clusters in Bangalore

Before applying configurations on Global Manager, ensure that below configurations are also applied in other location:

a. Transport Zones

b. IP Pools

c. Uplink Profiles

d. Compute Transport Node Profiles

e. Edge Transport Nodes config

f. Hosts are configured as host transport nodes.

g. RTEP configs on edge clusters in Delhi

BGP Setup

 

BGP Setup for Tier 0 Gateway Bangalore

BGP AS 65000 is used on Tier 0 Gateway Bangalore

e BGP is used between Tier 0 Gateway and upstream routers.

Physical network is under AS 65001

Traffic ingress and egress to/from subnet connected to Tier 1 Gateway Bangalore goes through physical routers in Bangalore.

This gives deterministic traffic flow.

AS Path prepending is used on physical routers of Delhi Location to influence this traffic flow.

Physical routers are sending a default route on a per BGP peer basis.

Routes from NSX are redistributed into BGP

BGP Setup for Tier 0 Gateway Delhi

 

BGP AS 65002 is used on Tier 0 Gateway Bangalore

e BGP is used between Tier 0 Gateway and upstream routers.

Physical network is under AS 65001

Traffic ingress and egress to/from subnet connected to Tier 1 Gateway Delhi goes through physical routers in Delhi

This gives deterministic traffic flow.

AS Path prepending is used on physical routers of Bangalore Location to influence this traffic flow.

Physical routers are sending a default route on a per BGP peer basis.

Routes from NSX are redistributed into BGP

 

Global Manager Configuration

Locations are added to Global Manager

Segments are created on Global Manager for uplink connectivity of Tier 0 Gateway

While creating segments on Global Manager, specify location, local transport zone and the VLAN ID

Tier 0 Gateway Bangalore

 

While creating stretched Tier 0 Gateway, specify the edge cluster and the corresponding location as Primary or Secondary.

For Tier 0 Gateway Bangalore, primary location is Bangalore.

For Tier 0 Gateway Delhi, primary location is Delhi

L3 interfaces on Tier 0 Gateway Bangalore

Likewise Tier 0 Gateway is created with Delhi as primary location & Bangalore as secondary location.

Tier 1 Gateway config on Global Manager

Next create stretched Tier 1 Gateway and connect to the already defined Tier 0 Gateway.

Tier 1 Gateways on Global Manager

 

 

Segments on Global Manager connected to Tier 1 Gateway

Next deploy VMs and connect them to appropriate segments.

 

 

VM connected to Overlay Network

Validation

Trace from router in Delhi to VM behind Tier 1 Gateway Bangalore goes through physical router of Bangalore

Trace from router in Bangalore to VM behind Tier 1 Gateway Delhi goes through physical router of Delhi

 

Trace from loopback of second physical router in Delhi to VM behind Tier 1 Gateway Bangalore goes through physical router 1 of Bangalore location

Trace from VM behind Tier 1 Gateway Bangalore towards loopback of physical router 2 in Delhi goes through physical router 2 of Bangalore location

 

Trace from VM in Delhi to loopback of physical router 1 in Bangalore goes through physical router of Delhi

Trace from loopback of physical router 2 in Bangalore to VM behind Tier 1 Gateway Delhi goes through physical router in Delhi

RTEP to RTEP tunnel is established

NSX-V to NSX-T Migration using Layer 2 Bridging

This blog will explore how we can migrate workloads which are on hosts prepared for NSX-V to hosts prepared for NSX-T using NSX-T Layer 2 Bridging.

Cluster Setup

In the lab setup, four hosts ESXi 1 up to 4 are prepared for NSX-T and the remaining four hosts ESXi5 up to ESXi 8 are prepared for NSX-V

NSX-T edges used for Layer 2 bridging are on NSX-V prepared hosts.

Logical Setup

The above is the logical setup used in this lab.

IP Addressing

Above shows the IP addressing used in the lab.

BGP AS Numbering

The above picture shows the BGP AS numbering used.

BGP peerings

The above diagram shows the BGP peerings.

e-BGP peerings between NSX and the physical network.

i BGP between NSX-V edges and Distributed Logical Router.

There is no routing protocol between Tier 1 Gateway of NSX-T and Tier 0 Gateway upstream.

During migration, traffic flow will be through NSX-V edges which means that:

1. You can prefer not to advertise connected subnets on the Tier 1 Gateway

2. Or to keep BGP disabled on Tier 0 Gateway.

NSX-V Setup

NSX-V Prepared Cluster

Above picture shows the four hosts prepared for NSX-V

NSX-V Edges and DLR

The required NSX-V edges and DLR have been deployed.

Workloads hosted on both clusters

VM on VXLAN

One VM Windows 10-2 is hosted on this NSX-V prepared cluster.

We need to make sure that security settings of the port group (corresponding to the VXLAN being bridged) are set accordingly.

• Set promiscuous mode on the portgroup.
• Allow forged transmit on the portgroup.

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.5/administration/GUID-F133B293-5DEA-4DC8-99DB-6EF004C8D8D7.html

Security settings of VXLAN backed port group

NSX-T Setup

Overlay Transport Zone defined using REST API Client

To ensure unique mac addresses are used on layer 3 interfaces of DLR and Tier 1 Gateway respectively, ensure overlay transport zone in NSX-T is defined as above with

"nested_nsx": true

Compute host transport nodes prepared for NSX-T

NSX-T Edges

nsx-edge-1 and nsx-edge-2 are NSX-T edges which are used for Layer 2 bridging.

These edges are placed on cluster prepared for NSX-V.

The remaining two edges nsx-edge-3 & nsx-edge-4 are used for Tier 0 Gateway.

Edge used for L2 Bridging

Fast-path interfaces fp-eth0 and fp-eth1 on the edges used for Layer 2 bridging are used for Geneve traffic, they are uplinked to a trunk port group on VDS used for NSX-V preparation. This way all NSX traffic stays on this VDS which is also used for NSX-V host preparation.

NSX-T Edge Clusters

Tier 0 Gateway

Tier 1 Gateway

NSX-T Segment connected to Tier 1 Gateway

Gateway set on NSX-T Segment

Validation of the setup

VM on NSX-T Segment

Reach ability between physical router loopback and both VMs

The above picture shows BGP peerings between the router and NSX-V edges.

At this stage, the BGP peerings between the physical routers and NSX-T edges are down/disabled.

Reason being that all workloads from NSX-V prepared hosts are not yet on NSX-T prepared hosts.

Reach ability between VM on VXLAN to loopback of physical router

Reach ability between VM on NSX-T segment to loopback of physical router

At this point, we know that Layer 2 bridging is working as intended and that the layer 2 bridge is forwarding traffic upstream.

Traffic flow from VM on NSX-T Segment to loopback interface of physical router

Migration

Now we will migrate the VM which is on NSX-V prepared cluster to NSX-T prepared cluster.

With this, both the workloads will then be on NSX-T prepared cluster.

At this point of time, we need to ensure that workloads have Tier 1 Gateway as their gateway.

We will ensure BGP peerings between physical routers and NSX-T edges are now all up.

And disable the BGP peerings between physical routers and NSX-V edges

Workloads migrated to NSX-T prepared cluster

Above picture shows that workloads have moved to NSX-T prepared cluster.

After migration, traffic flow from physical router to VMs on NSX-T segment

From the physical router, we validate that BGP peerings with NSX-T edges are now up and those with NSX-V edges are down.

Traffic now starts flowing through NSX-T edges.

VM on NSX-T Segment to loopback IP of physical router

VM on NSX-T segment to loopback IP of physical router

Traffic flow after migration