There are two articles in Microsoft documentation that contain the most important guidelines to configure routing over ExpressRoute:
Both are good articles and describe correct design patterns, but they were written a long time ago and most readers struggle to map those concepts to the current technologies available in Azure, most notably in Virtual WAN. There are two crucial VWAN improvements that are both a curse and a blessing:
- Virtual WAN has the concept of secured hubs, which makes extremely easy deploying firewalls into the data path. While this has helped many customers to increase their network security posture, these modern designs have no tolerance for asymmetric routing, since firewalls will drop packets that belong to a flow that hasn’t been seen before.
- Route maps in Virtual WAN allow to perform traffic engineering in a way that was not possible before, especially if you don’t have access to routing manipulation in your on-premises routing devices. Both documentation articles cited earlier make generic references to how to influence routing in on-premises routers, but give no concrete configuration directives (every on-premises network is different). The availability of route maps in Virtual WAN changes that.
What architecture are we speaking about?
First of all, let’s clarify why we do this. If you have two or more on-premises data center locations, you should pick ExpressRoute locations close to where your data centers stand today, and from there go to your Azure regions over Microsoft’s backbone. Something like this:
In the previous figure, ExpressRoute (ER) locations 1 and 2 are chosen so that they are in proximity to the locations of the onprem data centers 1 and 2. Then you can go from each ER location to each Azure region over the Azure backbone: “standard” ER circuits allow you to connect an ER circuit to other Azure regions in the same continent, “premium” ER circuits widen this reach to any region across the world.
Alternatively, some customers optimize this design with the so-called “squared” connectivity pattern where each ER circuit is only connected to one of the Azure regions, not to both:
This might be interesting because other than the “standard” and “premium” ER circuit types that we saw earlier, there is another type called “local”: it is much cheaper (although it has some limitations, read more about it here), since it only allows to connect ER circuits to VNets in one specific region. Note that there is a tension in this design, because aligning ER circuit locations to Azure regions might mean that you are not aligning them to onprem DC locations, as we recommended earlier. This tension disappears if all three levels are aligned to each other: for example, your onprem DC 1 might be Frankfurt, then you also choose “Frankfurt Metro” as the ER location 1, and “Germany West Central” as Azure region 1. Your onprem DC 2 might be Madrid, your ER location 2 “Madrid Metro” and your Azure region 2 “Spain Central”.
What if you only have 1 onprem location, but you would still like to have multiple Azure regions for resiliency, latency or capacity purposes? Typically you begin with a single, redundant ER location (meaning “Metro“) close to your onprem DC location:
Of course, if you have multiple Azure regions for resiliency purposes, it would be consistent to also expand your ExpressRoute locations for resiliency purposes:
The obvious follow-up question to the previous is why do you need such a high resiliency in the connection between Azure and onprem, if the onprem data center itself is not that resilient. However, there might be situations where this design is required.
The problem: asymmetric routing
In most of the designs above you have built redundant paths, and where there are redundant paths there can be asymmetrical routing. Since chances are that you will have firewalls in the Azure Virtual WAN hubs, asymmetrical routing means packet drops. How can we tune routing, so that traffic paths are predictable?
This is the test bed that I have tested (thanks to my colleague Mauricio Rojas for the idea):
This design doesn’t align 100% with all four of the designs we saw in the previous section, but the concepts about traffic engineering do. The main issue of this design is the existing routing asymmetry derived from the path redundancy. Think for example of a flow between spoke 1 and onprem:
- Spoke 1 will send traffic to hub1. Hub1 is connected to onprem via its own ExpressRoute circuit, so that is where packets will go.
- For the return traffic things look a bit different. The onprem router (simulated with a Google Cloud Router in my lab) sees identical prefixes coming from both ExpressRoute circuits, so it will pick one (an onprem router might do ECMP (Equal Cost MultiPathing) depending on the routing protocol and how it is configured).
If the onprem router happens to choose the “wrong” ExpressRoute circuit for the return traffic, the firewalls in Virtual WAN will drop the asymmetric flow as the following diagram shows:
We can follow how spoke 1’s prefix is advertised throughout the topology:
The crucial fact is that the MSEEs remove all BGP private ASNs (Autonomous System Numbers) before advertising the routes to the next hop (the Megaport cloud routers), because customers could potentially use those same ASNs in their network, since that is the idea behind private ASNs. However, the consequence is that the prepending of 65520 65520, which contained the information that the route is crossing a hub-to-hub link, is lost.
The fix: add your own prepending with route maps
Thanks to Virtual WAN route maps, you can add your own prepending to the routes before they get advertised to the MSEEs. Of course, you shouldn’t use private ASNs, because those would also be removed by the MSEEs when advertising to on-premises, but you have some options:
- You can use a public ASNs if you own one.
- You can use the reserved ASN 23456 (also called AS_TRANS), although personally I don’t think it is a great idea, since that ASN is reserved by RFC 4893 for migrations to 4-byte ASNs.
- Or you could use the reserved ASNs in the range 64496-64511, reserved as per RFC 5398 for documentation. This is the most common approach and my recommendation if you don’t own any public ASN.
When you prepend the routes in Virtual WAN with an ASN that will not be removed by the ExpressRoute MSEEs, the information about which routes are best to use for each destination will successfully arrive to the on-premises router, as the following diagram shows:
Extreme traffic engineering
I call it “extreme” because it is a somehow exotic design, but more common than what you might think. The goal here is selecting the same circuit for all flows, even if it introduces additional latency. The reason is usually cost optimization: maybe that circuit has been deployed with the Local SKU, and is consequently significantly cheaper for high-bandwidth flows. In any case, here is the topology that you would want to achieve:
The premise here is that the on-premises route should always pick the same circuit for all flows, but still have some redundancy to the second circuit so that ExpressRoute does not become a single-point-of-failure exist. You can control the direction from on-premises to Azure using many different techniques, BGP weights and local preference being the most common ones. However, if those are not available, you can also achieve the same with BGP: similarly to the previous section, you could prepend all routes that are advertised to on-premises with an outbound route map in Virtual WAN.
What about the path from Azure to onprem? So far these flows have always been taking the local ExpressRoute circuit, which is the default behavior of Virtual WAN. However, this can be changed with two configurations:
- Virtual hubs have a setting called Hub Routing Preference (HRP). Per default it is set to always prefer local (same hub) ExpressRoute connections, but you can change it to AS-path for a more BGP compliant routing decision.
- Now you can use inbound route maps so that the routes coming from MSEE2 are not preferred. You should at least prepend 3 times, to make sure that the prepending of 65520 65520 for inter-hub routes is not getting in the way.
You can do this prepending selectively only for some pairs of routes, but remember not to cause asymmetric routing or your firewalls will remind you in the most nasty way.
Conclusion
Hopefully I could show you how the concepts in Microsoft documentation map to modern Virtual WAN features, please do let me know if there are any points that I should have also included. Thanks for reading!
Cloudtrooper 