NSX-V to NSX-T Migration using Layer 2 Bridging

This blog will explore how we can migrate workloads which are on hosts prepared for NSX-V to hosts prepared for NSX-T using NSX-T Layer 2 Bridging.

Cluster Setup

In the lab setup, four hosts ESXi 1 up to 4 are prepared for NSX-T and the remaining four hosts ESXi5 up to ESXi 8 are prepared for NSX-V

NSX-T edges used for Layer 2 bridging are on NSX-V prepared hosts.

Logical Setup

The above is the logical setup used in this lab.

IP Addressing

Above shows the IP addressing used in the lab.

BGP AS Numbering

The above picture shows the BGP AS numbering used.

BGP peerings

The above diagram shows the BGP peerings.

e-BGP peerings between NSX and the physical network.

i BGP between NSX-V edges and Distributed Logical Router.

There is no routing protocol between Tier 1 Gateway of NSX-T and Tier 0 Gateway upstream.

During migration, traffic flow will be through NSX-V edges which means that:

1. You can prefer not to advertise connected subnets on the Tier 1 Gateway

2. Or to keep BGP disabled on Tier 0 Gateway.

NSX-V Setup

NSX-V Prepared Cluster

Above picture shows the four hosts prepared for NSX-V

NSX-V Edges and DLR

The required NSX-V edges and DLR have been deployed.

Workloads hosted on both clusters

VM on VXLAN

One VM Windows 10-2 is hosted on this NSX-V prepared cluster.

We need to make sure that security settings of the port group (corresponding to the VXLAN being bridged) are set accordingly.

• Set promiscuous mode on the portgroup.
• Allow forged transmit on the portgroup.

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.5/administration/GUID-F133B293-5DEA-4DC8-99DB-6EF004C8D8D7.html

Security settings of VXLAN backed port group

NSX-T Setup

Overlay Transport Zone defined using REST API Client

To ensure unique mac addresses are used on layer 3 interfaces of DLR and Tier 1 Gateway respectively, ensure overlay transport zone in NSX-T is defined as above with

"nested_nsx": true

Compute host transport nodes prepared for NSX-T

NSX-T Edges

nsx-edge-1 and nsx-edge-2 are NSX-T edges which are used for Layer 2 bridging.

These edges are placed on cluster prepared for NSX-V.

The remaining two edges nsx-edge-3 & nsx-edge-4 are used for Tier 0 Gateway.

Edge used for L2 Bridging

Fast-path interfaces fp-eth0 and fp-eth1 on the edges used for Layer 2 bridging are used for Geneve traffic, they are uplinked to a trunk port group on VDS used for NSX-V preparation. This way all NSX traffic stays on this VDS which is also used for NSX-V host preparation.

NSX-T Edge Clusters

Tier 0 Gateway

Tier 1 Gateway

NSX-T Segment connected to Tier 1 Gateway

Gateway set on NSX-T Segment

Validation of the setup

VM on NSX-T Segment

Reach ability between physical router loopback and both VMs

The above picture shows BGP peerings between the router and NSX-V edges.

At this stage, the BGP peerings between the physical routers and NSX-T edges are down/disabled.

Reason being that all workloads from NSX-V prepared hosts are not yet on NSX-T prepared hosts.

Reach ability between VM on VXLAN to loopback of physical router

Reach ability between VM on NSX-T segment to loopback of physical router

At this point, we know that Layer 2 bridging is working as intended and that the layer 2 bridge is forwarding traffic upstream.

Traffic flow from VM on NSX-T Segment to loopback interface of physical router

Migration

Now we will migrate the VM which is on NSX-V prepared cluster to NSX-T prepared cluster.

With this, both the workloads will then be on NSX-T prepared cluster.

At this point of time, we need to ensure that workloads have Tier 1 Gateway as their gateway.

We will ensure BGP peerings between physical routers and NSX-T edges are now all up.

And disable the BGP peerings between physical routers and NSX-V edges

Workloads migrated to NSX-T prepared cluster

Above picture shows that workloads have moved to NSX-T prepared cluster.

After migration, traffic flow from physical router to VMs on NSX-T segment

From the physical router, we validate that BGP peerings with NSX-T edges are now up and those with NSX-V edges are down.

Traffic now starts flowing through NSX-T edges.

VM on NSX-T Segment to loopback IP of physical router

VM on NSX-T segment to loopback IP of physical router

Traffic flow after migration

OSPF Routing Protocol in NSX

OSPF NSX

The above topology illustrates OSPF Totally NSSA setup in NSX domain.

Distributed logical router is handling east-west traffic within the data center while the edges E1 and E2 are handling north-south traffic.

Edges E1 and E2 are configured in ECMP mode which allows more bandwidth to flow in north-south direction.

Upto eight edges E1 – E8 can be configured in ECMP mode to handle north-south traffic.

The other mode for edge is Active/Standby which can be utilized when stateful services like NAT, firewall, load balancer, SSL and IPSec VPN are to be configured on the edge.

In our topology above, such stateful services are not configured and the requirement is to have more bandwidth in north-south direction.

Downstream to the DLR are the logical switches corresponding to Web, App and DB tiers respectively.

And the workloads are connected to the logical switches.

A brief about DLR control VM.

DLR Control VM is required when:

a.       There are dynamic routing protocols configured

b.       Or when there is L2 Bridging configured.

L2 bridging is useful while migrating workloads from physical to virtual environment.

L2 bridging also facilitates extending physical network to virtual network and vice versa.

Referencing the topology and the routing protocol setup, a DLR control VM is required in this setup.

Also, when you deploy edge as a Distributed Logical Router, these two IP addresses need to be configured aside from IP addresses related to LIFs

-          Protocol address – This IP address corresponds to the control plane exchange.

-          Forwarding address which corresponds to data plane and this IP address handles exchange of data traffic.

===========================================================================

Routing setup:

a.       Totally NSSA area is configured which is corresponding to the NSX domain.

b.       With Totally NSSA area configuration, the ABRs will inject a default route into Totally NSSA area automatically.

c.       Totally NSSA area configuration will allow redistribution of connected interfaces on the DLR.

d.       Physical routers have been configured as OSPF Area Border Routers

So the physical routers are interfacing backbone area and the Totally NSSA Area.

e.       The DLR along with edges E1 and E2 are within Totally NSSA Area.

f.        NSX domain needs a default route for egress traffic from NSX domain.

Routes which do not belong to the Totally NSSA area and which are external to this area will be replaced by Type 3 default route.

g.       Edge E1 will have OSPF peering with DLR control VM and with physical routers upstream.

Likewise for edge E2.

h.       Two external VLANs are used from each edge to peer with the physical routers.

The number of external VLANs will be equal to number of VDS uplinks. Each external VLAN will be carried on one ESXi uplink.

With such a setup, both uplinks from ESXi will be utilized for forwarding data traffic. Also a failure of one of the physical uplinks will cause data traffic to shift to the other physical uplink from ESXi.