Data Center Technologies: January 2019

Cross Vcenter NSX using Local Egress

The above topology has been used for the purpose of this lab.

In this blog post, we are going to cover Active Active Cross Vcenter NSX.
BGP is used as a routing protocol in this setup above.
And appropriate BGP peerings have been illustrated in topology above.
We are using eBGP between NSX edge and physical router.
iBGP is used within NSX between NSX edge and UDLR.
Notice the connectivity using Transit LS in both sites.
A separate Logical Switch is used in Site 2 between UDLR and NSX edge.

In my previous blogs on Cross Vcenter NSX using BGP and Cross Vcenter NSX using OSPF, we did not cover local egress feature of UDLR.

With local egress enabled, UDLR Control VMs are deployed in both the sites - Site 1 and Site 2.
There's a parameter called Locale ID which controls the routes being advertised to the hosts in each site.
Site specific Locale ID is associated with the routes.

Routes learnt from edges by the UDLR are sent to the Universal Controller Cluster and from there on to the hosts.

With Locale ID, hosts in Site A will forward egress traffic towards the NSX edges in Site A.
Likewise for Site B.

Traffic sourced from the physical network and destined for subnets behind UDLR (UDLR being deployed using local egress) can reach the UDLR via Site A NSX edge or via Site B NSX edge.

Essentially, asymmetric routing takes place and there is no deterministic traffic flow.
Also, if you have firewalls upstream to the NSX edges in each of the sites, then the firewall will drop such asymmetric traffic.
Firewall can receive SYN-ACK when it has not seen the SYN packet for a specific TCP session.
So one should be careful and not use Local Egress in such a situation where there are firewalls upstream to NSX edges.

Now let's see how a UDLR is deployed using Local Egress.

The important thing when you want to enable Local Egress on the UDLR is to ensure that we select this option while deploying UDLR itself.

Notice that only Site A Data Center is listed here.

Select cluster and datastore

We will do the interface configurations later.

Click on Finish to start deploying the UDLR.

Verify that Local Egress is enabled on the UDLR

=======================================================================

Routing configurations of Site A

Below are the interface configurations of UDLR.
Routing configurations are applied as per our topology above.
1. BGP AS number 65001 is used for Site A and BGP AS 65011 is used for Site B.
2. We will see from interface configurations of UDLR in Site B that we are unable to edit the interface configurations there.


Interface Configurations of UDLR under Primary NSX Manager

UDLR Control VM of Site A

UDLR Control VM will be deployed in both the sites.

========================================================================

Routing configs of Site B

Unable to edit interface configurations of UDLR from Secondary NSX Manager

UDLR Control VM is deployed in Site B as well

We have covered cross vcenter NSX with OSPF previously.
Here we cover cross vcenter NSX with BGP.
In this above topology:

There are two vcenters, one in the primary site and the other in DR site.
Primary NSX manager is deployed in primary site.
Secondary NSX manager is deployed in DR site.
Primary NSX manager is linked with vcenter in primary site.
Secondary NSX manager is linked with vcenter in DR site.
Universal controller cluster is deployed under vcenter 1
Cross vcenter NSX is configured using the primary and secondary NSX managers.
Universal transport zone is defined which will span the clusters across both sites. We are able to define universal logical switches using this single universal transport zone. Keep in mind that we are able to deploy only one universal transport zone. And all universal objects are only defined from primary NSX manager.
Two NSX edges are deployed in primary site and two NSX edges are deployed in DR site.
This is a active/passive model and we will ensure ingress/egress traffic from the primary site. This is a preferred setup since it gives deterministic traffic flow and keeps the traffic flow symmetrical, ingress/egress via a particular site.
To influence inbound traffic, we will leverage BGP AS Path Prepending in the physical network.

BGP Configuration:
As depicted in topology above:

iBGP is used within NSX domain. UDLR has four iBGP peerings with the NSX edges of both sites.
Single private AS 64512 is used within NSX domain
BGP AS 1 is used in the physical network upstream.
eBGP is used between NSX edges and the physical routers
iBGP is used between UDLR and NSX edges.
We will modify BGP weight on UDLR to influence outbound traffic.
NSX domain is like a stub network. Hence there is no hard requirement of full mesh iBGP within this NSX domain.
We are going to redistribute connected networks on UDLRs and the edges.
Default route will be sent by the physical routers towards NSX edges only through the eBGP peering. This configuration is covered in the BGP configurations below.

BGP Weight Modification on UDLR

Redistribute connected on UDLR

Redistribute connected on the NSX Edge

iBGP peerings on UDLR

Default route on UDLR

Now let's also map this topology with Cisco routers and lab this up.
I've added UDLR2 as well in this topology.
Ingress/egress for subnets behind UDLR2 will be via DR site. This is covered in the traffic flow section.
BGP setup for Cross Vcenter NSX using Cisco routers

This topology is for demo purpose only.
Core 1 and Core 2 routers in topology above are single point of failures. Also the route reflector configuration in this topology has single point of failures. Ideally redundant route reflectors should be deployed for exchange of route updates.
Appropriate measures should be taken while designing production networks to avoid such single point of failures.

This topology is essentially utilized to cover the BGP configs relevant to NSX.
The appropriate BGP peerings are depicted in the diagram above.
Core 1 is configured as route-reflector for physical routers in primary site.
Core 2 is configured as route-reflector for physical routers in DR site.
Core 1 and Core 2 have i BGP peering between them.
OSPF is utilized as IGP in BGP AS 1
Physical routers are sending a default route towards NSX edges only.
AS Path Prepending has been configured on physical routers of DR site for subnets behind UDLR1. This will ensure ingress/egress traffic for subnets behind UDLR1 is via primary site.
For subnets behind UDLR2, AS Path prepending will be done for these subnets in primary site. Ingress/egress traffic flow towards subnets behind UDLR2 will be via DR site.

Important notes about updates in a route-reflector setup:

Update received from a route-reflector client will be send across to other clients, non-clients and eBGP peers
Update received from a non client will be send across to other clients and eBGP peers only and such update will not be sent to other non clients. In the topology above, Core 2 is a non client for Core 1. Also Core 1 is a non client for Core 2.
Update received from a e BGP peer will be sent across to clients, non clients and other e BGP peers.

Router Configurations