Cross Vcenter NSX with OSPF

References:

Cross Vcenter NSX Design Guide

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmware-multi-site-solutions-cross-vcenter-nsx-design-guide.pdf 

 

Intra area routes versus inter area routes

https://blog.ipspace.net/2008/01/e1-and-e2-routes-in-ospf.html

 
In one of the earlier posts, we have covered running OSPF in NSX domain.

Here we try to cover running OSPF in a Cross Vcenter NSX setup.

In the above topology:
1. There are two NSX managers.
Primary NSX manager in site 1
And secondary NSX manager in site 2
This is a Active/Passive model.

2. One vcenter in site 1 and the other in site 2

3. Universal Distributed Logical Router UDLR is connected to edges in primary & secondary site using a single Transit logical switch.

4. There are two edges in site 1 and additional two edges in site 2

5. The edges are connected to physical routers as shown above.

6. The physical routers are OSPF Area Border Routers interfacing with OSPF backbone area 0.

7. Totally NSSA area configuration is applied in this setup.
The Area Border Routers are injecting a default route automatically into the NSX domain.

8. Only one Universal Transport Zone can be created to span both the sites.

9. Universal Distributed Logical Router UDLR spans both the sites and allows connecting workloads from both the sites.
The status of UDLR in primary site is 'deployed' while in the secondary site it is 'active'.

10. Universal Distributed Firewall spans both the sites.

11. Universal Controller Cluster is deployed in Site 1

12. Universal objects can be created only from the Primary NSX Manager.
If there is a need to deploy additional UDLR, then this has to be done from Primary NSX Manager only. In this case the UDLR will consume resources (compute, memory and storage) of Site 1

13. UDLR control VM will reside in Site 1 and will establish OSPF peering with NSX edges from both sites 1 and 2.

14. Site 1 is preferred for ingress/egress traffic because the OSPF cost is increased for links between Site 2 physical routers and Site 2 NSX edges.

More traffic flows are discussed in traffic flow section below.


 ===============================================

Let's map this NSX topology with Cisco routers and explore the routing tables on devices.

Subnet 192.168.1.0/24 is connected to router UDLR.

And we check the route lookup for 192.168.1.0 on routers:
1. Core2
2. Core1
3. Physical Router 1 of Site 2
4. Edges of site 1 and site 2

Note the route lookup for 192.168.1.0/24 on Site 2 router 1 prefers the route towards NSX edges in the same site, site 2.
This is despite increasing the OSPF cost on the links connecting Site 2 physical routers and Site 2 NSX edges.
This is because OSPF prefers intra area routes as compared to inter area routes.

Above output shows that OSPF metric is increased to 120 via Site 2 physical routers.

Route for 192.168.1.0/24 on Core2

 Next hop from Core 2 for subnet 192.168.1.0/24 is Core 1 because of lower OSPF cost between site 1 physical routers and Site 1 NSX edges.

Route for 192.168.1.0/24 on Core 1

Route for 192.168.1.0/24 on Site2 Router 1

Route for 192.168.1.0/24 on Site 1 Edge

Route for 192.168.1.0/24 on Site2 Edge

Default route on UDLR

==================================================

Traffic flows

UDLR follows the default route

If there are other services/customers connected to the physical routers of Site 2, then such traffic will not go via Site 1. Instead be routed towards local NSX edges in Site 2 because intra area routes are preferred as compared to inter area routes.

Ingress from Core WAN to App Tier LS

 

OSPF Routing Protocol in NSX

OSPF NSX

The above topology illustrates OSPF Totally NSSA setup in NSX domain.

Distributed logical router is handling east-west traffic within the data center while the edges E1 and E2 are handling north-south traffic.

Edges E1 and E2 are configured in ECMP mode which allows more bandwidth to flow in north-south direction.

Upto eight edges E1 – E8 can be configured in ECMP mode to handle north-south traffic.

The other mode for edge is Active/Standby which can be utilized when stateful services like NAT, firewall, load balancer, SSL and IPSec VPN are to be configured on the edge.

In our topology above, such stateful services are not configured and the requirement is to have more bandwidth in north-south direction.

Downstream to the DLR are the logical switches corresponding to Web, App and DB tiers respectively.

And the workloads are connected to the logical switches.

A brief about DLR control VM.

DLR Control VM is required when:

a.       There are dynamic routing protocols configured

b.       Or when there is L2 Bridging configured.

L2 bridging is useful while migrating workloads from physical to virtual environment.

L2 bridging also facilitates extending physical network to virtual network and vice versa.

Referencing the topology and the routing protocol setup, a DLR control VM is required in this setup.

Also, when you deploy edge as a Distributed Logical Router, these two IP addresses need to be configured aside from IP addresses related to LIFs

-          Protocol address – This IP address corresponds to the control plane exchange.

-          Forwarding address which corresponds to data plane and this IP address handles exchange of data traffic.

===========================================================================

Routing setup:

a.       Totally NSSA area is configured which is corresponding to the NSX domain.

b.       With Totally NSSA area configuration, the ABRs will inject a default route into Totally NSSA area automatically.

c.       Totally NSSA area configuration will allow redistribution of connected interfaces on the DLR.

d.       Physical routers have been configured as OSPF Area Border Routers

So the physical routers are interfacing backbone area and the Totally NSSA Area.

e.       The DLR along with edges E1 and E2 are within Totally NSSA Area.

f.        NSX domain needs a default route for egress traffic from NSX domain.

Routes which do not belong to the Totally NSSA area and which are external to this area will be replaced by Type 3 default route.

g.       Edge E1 will have OSPF peering with DLR control VM and with physical routers upstream.

Likewise for edge E2.

h.       Two external VLANs are used from each edge to peer with the physical routers.

The number of external VLANs will be equal to number of VDS uplinks. Each external VLAN will be carried on one ESXi uplink.

With such a setup, both uplinks from ESXi will be utilized for forwarding data traffic. Also a failure of one of the physical uplinks will cause data traffic to shift to the other physical uplink from ESXi.