Cross Vcenter NSX using Local Egress (Active-Active)

Cross Vcenter NSX using Local Egress

The above topology has been used for the purpose of this lab.

In this blog post, we are going to cover Active Active Cross Vcenter NSX.
BGP is used as a routing protocol in this setup above.
And appropriate BGP peerings have been illustrated in topology above.
We are using eBGP between NSX edge and physical router.
iBGP is used within NSX between NSX edge and UDLR.
Notice the connectivity using Transit LS in both sites.
A separate Logical Switch is used in Site 2 between UDLR and NSX edge.

In my previous blogs on Cross Vcenter NSX using BGP and Cross Vcenter NSX using OSPF, we did not cover local egress feature of UDLR.

With local egress enabled, UDLR Control VMs are deployed in both the sites - Site 1 and Site 2.
There's a parameter called Locale ID which controls the routes being advertised to the hosts in each site.
Site specific Locale ID is associated with the routes.

Routes learnt from edges by the UDLR are sent to the Universal Controller Cluster and from there on to the hosts.

With Locale ID, hosts in Site A will forward egress traffic towards the NSX edges in Site A.
Likewise for Site B.

Traffic sourced from the physical network and destined for subnets behind UDLR (UDLR being deployed using local egress) can reach the UDLR via Site A NSX edge or via Site B NSX edge.

Essentially, asymmetric routing takes place and there is no deterministic traffic flow.
Also, if you have firewalls upstream to the NSX edges in each of the sites, then the firewall will drop such asymmetric traffic.
Firewall can receive SYN-ACK when it has not seen the SYN packet for a specific TCP session.
So one should be careful and not use Local Egress in such a situation where there are firewalls upstream to NSX edges.



Now let's see how a UDLR is deployed using Local Egress.

The important thing when you want to enable Local Egress on the UDLR is to ensure that we select this option while deploying UDLR itself.

 

Notice that only Site A Data Center is listed here.



Select cluster and datastore

We will do the interface configurations later.

Click on Finish to start deploying the UDLR.

 Verify that Local Egress is enabled on the UDLR

=======================================================================

Routing configurations of Site A

Below are the interface configurations of UDLR.
Routing configurations are applied as per our topology above.
1. BGP AS number 65001 is used for Site A and BGP AS 65011 is used for Site B.
2. We will see from interface configurations of UDLR in Site B that we are unable to edit the interface configurations there.

Interface Configurations of UDLR under Primary NSX Manager

UDLR Control VM of Site A

 UDLR Control VM will be deployed in both the sites.

========================================================================

Routing configs of Site B

Unable to edit interface configurations of UDLR from Secondary NSX Manager

UDLR Control VM is deployed in Site B as well

Cross Vcenter NSX with BGP

We have covered cross vcenter NSX with OSPF previously.
Here we cover cross vcenter NSX with BGP.
In this above topology:

1. There are two vcenters, one in the primary site and the other in DR site.
2. Primary NSX manager is deployed in primary site.
3. Secondary NSX manager is deployed in DR site.
4. Primary NSX manager is linked with vcenter in primary site.
5. Secondary NSX manager is linked with vcenter in DR site.
6. Universal controller cluster is deployed under vcenter 1
7. Cross vcenter NSX is configured using the primary and secondary NSX managers.
8. Universal transport zone is defined which will span the clusters across both sites. We are able to define universal logical switches using this single universal transport zone. Keep in mind that we are able to deploy only one universal transport zone. And all universal objects are only defined from primary NSX manager.
9.  Two NSX edges are deployed in primary site and two NSX edges are deployed in DR site.
10.  This is a active/passive model and we will ensure ingress/egress traffic from the primary site. This is a preferred setup since it gives deterministic traffic flow and keeps the traffic flow symmetrical, ingress/egress via a particular site.
11. To influence inbound traffic, we will leverage BGP AS Path Prepending in the physical network.

BGP Configuration:
As depicted in topology above:

1. iBGP is used within NSX domain. UDLR has four iBGP peerings with the NSX edges of both sites.
2. Single private AS 64512 is used within NSX domain
3. BGP AS 1 is used in the physical network upstream.
4.  eBGP is used between NSX edges and the physical routers
5.  iBGP is used between UDLR and NSX edges.
6.  We will modify BGP weight on UDLR to influence outbound traffic.
7. NSX domain is like a stub network. Hence there is no hard requirement of full mesh iBGP within this NSX domain.
8. We are going to redistribute connected networks on UDLRs and the edges.
9.  Default route will be sent by the physical routers towards NSX edges only through the eBGP peering. This configuration is covered in the BGP configurations below.

BGP Weight Modification on UDLR

Redistribute connected on UDLR

Redistribute connected on the NSX Edge



iBGP peerings on UDLR


Default route on UDLR

---

Now let's also map this topology with Cisco routers and lab this up.
I've added UDLR2 as well in this topology.
Ingress/egress for subnets behind UDLR2 will be via DR site. This is covered in the traffic flow section.
BGP setup for Cross Vcenter NSX using Cisco routers

This topology is for demo purpose only.
Core 1 and Core 2 routers in topology above are single point of failures. Also the route reflector configuration in this topology has single point of failures. Ideally redundant route reflectors should be deployed for exchange of route updates.
Appropriate measures should be taken while designing production networks to avoid such single point of failures.

1. This topology is essentially utilized to cover the BGP configs relevant to NSX.
2. The appropriate BGP peerings are depicted in the diagram above.
3. Core 1 is configured as route-reflector for physical routers in primary site.
4. Core 2 is configured as route-reflector for physical routers in DR site.
5. Core 1 and Core 2 have i BGP peering between them.
6. OSPF is utilized as IGP in BGP AS 1
7.  Physical routers are sending a default route towards NSX edges only.
8. AS Path Prepending has been configured on physical routers of DR site for subnets behind UDLR1. This will ensure ingress/egress traffic for subnets behind UDLR1 is via primary site.
9. For subnets behind UDLR2, AS Path prepending will be done for these subnets in primary site. Ingress/egress traffic flow towards subnets behind UDLR2 will be via DR site.

Important notes about updates in a route-reflector setup:

1. Update received from a route-reflector client will be send across to other clients, non-clients and eBGP peers
2. Update received from a non client will be send across to other clients and eBGP peers only and such update will not be sent to other non clients. In the topology above, Core 2 is a non client for Core 1. Also Core 1 is a non client for Core 2.
3. Update received from a e BGP peer will be sent across to clients, non clients and other e BGP peers.

---

Router Configurations


Site 1 Router 1 Config


Site 1 Router 2 Config


Site 2 Router 1 Configuration




Site 2 Router 2 Configuration

Route lookup for 155.1.10.0/24 on Site 2 Router 2



Route lookup for 155.1.10.0/24 on Core 2
Notice only the best path is sent across Site2-Router2 towards Core2

---

Traffic Flows

Traffic flow towards subnets behind UDLR1

Traffic flow towards subnets behind UDLR2

Cross Vcenter NSX with OSPF

References:

Cross Vcenter NSX Design Guide

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmware-multi-site-solutions-cross-vcenter-nsx-design-guide.pdf 

 

Intra area routes versus inter area routes

https://blog.ipspace.net/2008/01/e1-and-e2-routes-in-ospf.html

 
In one of the earlier posts, we have covered running OSPF in NSX domain.

Here we try to cover running OSPF in a Cross Vcenter NSX setup.

In the above topology:
1. There are two NSX managers.
Primary NSX manager in site 1
And secondary NSX manager in site 2
This is a Active/Passive model.

2. One vcenter in site 1 and the other in site 2

3. Universal Distributed Logical Router UDLR is connected to edges in primary & secondary site using a single Transit logical switch.

4. There are two edges in site 1 and additional two edges in site 2

5. The edges are connected to physical routers as shown above.

6. The physical routers are OSPF Area Border Routers interfacing with OSPF backbone area 0.

7. Totally NSSA area configuration is applied in this setup.
The Area Border Routers are injecting a default route automatically into the NSX domain.

8. Only one Universal Transport Zone can be created to span both the sites.

9. Universal Distributed Logical Router UDLR spans both the sites and allows connecting workloads from both the sites.
The status of UDLR in primary site is 'deployed' while in the secondary site it is 'active'.

10. Universal Distributed Firewall spans both the sites.

11. Universal Controller Cluster is deployed in Site 1

12. Universal objects can be created only from the Primary NSX Manager.
If there is a need to deploy additional UDLR, then this has to be done from Primary NSX Manager only. In this case the UDLR will consume resources (compute, memory and storage) of Site 1

13. UDLR control VM will reside in Site 1 and will establish OSPF peering with NSX edges from both sites 1 and 2.

14. Site 1 is preferred for ingress/egress traffic because the OSPF cost is increased for links between Site 2 physical routers and Site 2 NSX edges.

More traffic flows are discussed in traffic flow section below.


 ===============================================

Let's map this NSX topology with Cisco routers and explore the routing tables on devices.

Subnet 192.168.1.0/24 is connected to router UDLR.

And we check the route lookup for 192.168.1.0 on routers:
1. Core2
2. Core1
3. Physical Router 1 of Site 2
4. Edges of site 1 and site 2

Note the route lookup for 192.168.1.0/24 on Site 2 router 1 prefers the route towards NSX edges in the same site, site 2.
This is despite increasing the OSPF cost on the links connecting Site 2 physical routers and Site 2 NSX edges.
This is because OSPF prefers intra area routes as compared to inter area routes.

Above output shows that OSPF metric is increased to 120 via Site 2 physical routers.

Route for 192.168.1.0/24 on Core2

 Next hop from Core 2 for subnet 192.168.1.0/24 is Core 1 because of lower OSPF cost between site 1 physical routers and Site 1 NSX edges.

Route for 192.168.1.0/24 on Core 1

Route for 192.168.1.0/24 on Site2 Router 1

Route for 192.168.1.0/24 on Site 1 Edge

Route for 192.168.1.0/24 on Site2 Edge

Default route on UDLR

==================================================

Traffic flows

UDLR follows the default route

If there are other services/customers connected to the physical routers of Site 2, then such traffic will not go via Site 1. Instead be routed towards local NSX edges in Site 2 because intra area routes are preferred as compared to inter area routes.

Ingress from Core WAN to App Tier LS

 

Vrealize Network Insight 3.8 Installation

References:

VRNI Architecture:

https://blogs.vmware.com/management/2016/10/vrealize-network-insight-3-1-scalability.html

Installation Guide:

https://docs.vmware.com/en/VMware-vRealize-Network-Insight/3.8/vRealize-Network-Insight-Installation-Guide.pdf

This installation is broken down into two parts:

- One for platform &

- The other for Collector aka Proxy.

During the deployment of OVA for Platform & Proxy, you will use Thin Provision as the virtual disk format.

1. The first step with regards to Vrealize Network Insight installation is to deploy the OVA for Platform & initialize the Platform.

Once deployed, power on the VM and get to the console access of the platform VM.

Here run the setup command.

 And enter the passwords for SSH user and CLI user.

As shown in the picture above, enter parameters related to network configuration:
- IP address
- Netmask
- Default gateway
- DNS
- & enter the name for domain search

Enter NTP server IP address.

Next, we will open up a browser and go to https://vrni_platform_ip 
In my case, it is https://192.168.110.99
Enter the license keys
And click on validate.

Once licensed and validated, you will next generate shared secret which will be used for Collector aka Proxy VM.

2.Collector aka Proxy deployment & initialization.

When you deploy the OVA for Proxy, you will need the shared secret as outlined in the previous step.

Next, get to the console of Proxy VM and initialize as outlined below.

You can now login to the Platform and enter credentials.