You probably know Azure Virtual WAN: it is an Azure service that provides any-to-any connectivity across regions out of the box, or a “global transit network architecture”, as they describe here:
Essentially Virtual WAN is a set of Microsoft-managed virtual hubs peered to each other, where you would connect your VNets and/or branches (ExpressRoute, Site-to-Site and Point-to-Site). One area that was a bit gray until now was about how to influence routing between hubs if there is more than one route, which is now configurable with the new and shiny Virtual Hub Routing Preference (in preview at the time of this writing).
In this blog I am going to dive deeper on how it works, and what effects does it have.
The test lab
I will start with this test bed: two virtual hubs, a spoke VNet peered to each virtual hub, and two ExpressRoute circuits each connected to both virtual hubs. This is a very common topology used by many customers:
ExpressRoute routes are preferred by default
Let’s have a peak into hub1’s effective routes for the default route table (I am not using any custom routing, all connections are associated and propagating to the default route table):
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix ASPath NextHopType NextHop Origin
-------------- ----------------- -------------------------- ------------------------------------------------------ -----------------------------------------------------
10.225.0.0/24 12076-65001-16550 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.1.1.0/24 Virtual Network Connection virtualHubs/hub1/hubVirtualNetworkConnections/spoke11 virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
10.2.1.0/24 12076-12076 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
192.168.2.0/23 12076-12076 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.226.0.0/24 12076-65001-16550 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
By the way, don’t worry about the awk fanciness of the previous command, I just do it to make it a bit more readable.
If you have a look at the highlighted route for 10.2.1.0/24, you will see that the next hop is ExpressRoute. Why? Well, that is documented in the Virtual WAN FAQ and in that document describing the new Hub Routing Preference: in the presence of two routes with the same prefix and length, Virtual WAN will prefer the one coming from ExpressRoute. So what does the other route look like? We can have a look at the effective routes in the ExpressRoute connection:
❯ az network vhub get-effective-routes --resource-type ExpressRouteConnection --resource-id $hub1_ergw_cx0_id -g $rg -n hub1 -o table --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix ASPath NextHopType NextHop Origin
-------------- ----------------------------- -------------------------- ------------------------------------------------------ -----------------------------------------------------
10.225.0.0/24 12076-65001-16550 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.225.0.0/24 65520-65520-12076-65001-16550 Remote Hub virtualHubs/hub2 virtualHubs/hub2
10.1.1.0/24 Virtual Network Connection virtualHubs/hub1/hubVirtualNetworkConnections/spoke11 virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
10.1.1.0/24 65520-65520-12076-12076 Remote Hub virtualHubs/hub2 virtualHubs/hub2
10.2.1.0/24 12076-12076 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.2.1.0/24 65520-65520 Remote Hub virtualHubs/hub2 virtualHubs/hub2
192.168.2.0/23 12076-12076 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.226.0.0/24 12076-65001-16550 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.226.0.0/24 65520-65520-12076-65001-16550 Remote Hub virtualHubs/hub2 virtualHubs/hub2
Interesting, here we do see both routes: one coming from ExpressRoute, where the ExpressRoute MSEE is reflecting the routes coming from hub2 back to hub1, and the other one directly from hub2. Let’s verify that hub1’s routing preference is set to ExpressRoute:
❯ az network vhub list -g $rg -o table AddressPrefix AllowBranchToBranchTraffic HubRoutingPreference Location Name PreferredRoutingGateway ProvisioningState ResourceGroup RoutingState VirtualRouterAsn --------------- ---------------------------- ---------------------- ------------------ ------ ------------------------- ------------------- --------------- -------------- ------------------ 192.168.0.0/23 False ExpressRoute germanywestcentral hub1 ExpressRoute Succeeded vwaneastus Provisioned 65515 192.168.2.0/23 False ExpressRoute westcentralus hub2 ExpressRoute Succeeded vwaneastus Provisioned 65515
Note that this will be hard to find out with our old friend traceroute testing connectivity between the virtual machines in each region, since some of the components in Virtual WAN and ExpressRoute do not react friendly to TTLs expired in transit:
jose@spoke11-vm:~$ traceroute 10.2.1.4 traceroute to 10.2.1.4 (10.2.1.4), 64 hops max 1 192.168.0.4 3.141ms 1.315ms 1.305ms 2 * * * 3 10.2.1.4 137.564ms 134.420ms 133.963ms
From this traceroute it is impossible telling if the next hop for our spoke VM is Virtual WAN or the ExpressRoute infrastructure.
Switching to ASPath preference
Changing the Hub Routing Preference to ASPath is pretty easy:
❯ routing_preference=ASPath # Can be ASPath, ExpressRoute or VpnGateway ❯ az network vhub update -n hub1 -g $rg --hub-routing-preference $routing_preference -o none ❯ az network vhub update -n hub2 -g $rg --hub-routing-preference $routing_preference -o none ❯ az network vhub list -o table AddressPrefix AllowBranchToBranchTraffic HubRoutingPreference Location Name PreferredRoutingGateway ProvisioningState ResourceGroup RoutingState VirtualRouterAsn --------------- ---------------------------- ---------------------- ------------------ ------ ------------------------- ------------------- --------------- -------------- ------------------ 192.168.2.0/23 False ASPath westcentralus hub2 ExpressRoute Succeeded vwaneastus Provisioned 65515 192.168.0.0/23 False ASPath germanywestcentral hub1 ExpressRoute Succeeded vwaneastus Provisioned 65515
Let’s check again the effective route tables in hub1’s default route table:
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix ASPath NextHopType NextHop Origin
-------------- ----------------- -------------------------- ------------------------------------------------------ ------------------------------------------------------
10.225.0.0/24 12076-65001-16550 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.1.1.0/24 Virtual Network Connection virtualHubs/hub1/hubVirtualNetworkConnections/spoke11 virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
192.168.2.0/23 12076-12076-12076 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.226.0.0/24 12076-65001-16550 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.2.1.0/24 65520-65520 Remote Hub virtualHubs/hub2 virtualHubs/hub2
There you go! Now Virtual WAN will send spoke-to-spoke traffic directly through the connection between the hubs, without hitting ExpressRoute.
BGP from the VNet
What about when you have an “indirect spoke” topology with BGP routes injected from a “direct spoke”? Let’s add those to our test lab:
With the Hub Routing Preference set to ExpressRoute here is what the default route table looks like:
❯ az network vhub list -o table
AddressPrefix AllowBranchToBranchTraffic HubRoutingPreference Location Name PreferredRoutingGateway ProvisioningState ResourceGroup RoutingState VirtualRouterAsn
--------------- ---------------------------- ---------------------- ------------------ ------ ------------------------- ------------------- --------------- -------------- ------------------
192.168.2.0/23 False ExpressRoute westcentralus hub2 ExpressRoute Succeeded vwaneastus Provisioned 65515
192.168.0.0/23 False ExpressRoute germanywestcentral hub1 ExpressRoute Succeeded vwaneastus Provisioned 65515
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix NextHopType NextHop Origin ASPath
---------------- -------------------------- ------------------------------------------------------ ------------------------------------------------------ -----------------
10.225.0.0/24 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw 12076-65001-16550
192.168.2.0/23 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw 12076-12076
10.226.0.0/24 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw 12076-65001-16550
10.1.1.0/24 Virtual Network Connection virtualHubs/hub1/hubVirtualNetworkConnections/spoke11 virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
10.2.1.0/24 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw 12076-12076
10.1.2.0/24 Virtual Network Connection virtualHubs/hub1/hubVirtualNetworkConnections/spoke12 virtualHubs/hub1/hubVirtualNetworkConnections/spoke12
10.2.2.0/24 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw 12076-12076
10.1.22.0/24 HubBgpConnection virtualHubs/hub1/bgpConnections/spoke12 virtualHubs/hub1/bgpConnections/spoke12 65012
10.1.21.0/24 HubBgpConnection virtualHubs/hub1/bgpConnections/spoke12 virtualHubs/hub1/bgpConnections/spoke12 65012
168.63.129.16/32 HubBgpConnection virtualHubs/hub1/bgpConnections/spoke12 virtualHubs/hub1/bgpConnections/spoke12 65012
10.2.21.0/24 Remote Hub virtualHubs/hub2 virtualHubs/hub2 65520-65520-65012
10.2.22.0/24 Remote Hub virtualHubs/hub2 virtualHubs/hub2 65520-65520-65012
Now this is interesting: the remote “direct” spoke (10.2.2.0/24) is known both from ExpressRoute and the inter-hub link, so since our routing preference is ExpressRoute, we prefer that one. However, it looks like the remote “indirect” spokes are not known via ExpressRoute at all, so even with routing preference of ExpressRoute the inter-hub route is chosen (there is not any other one).
If you are wondering why ExpressRoute would only reflect routes coming from connections but not routes coming from BGP peerings, I am afraid I don’t have an answer for you.
And as we would expect, after switching the routing preference to ASPath all remote spokes (direct and indirect) go over the inter-hub link:
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix ASPath NextHopType NextHop Origin
---------------- ----------------- -------------------------- --------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------
10.225.0.0/24 12076-65001-16550 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
192.168.2.0/23 12076-12076-12076 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.226.0.0/24 12076-65001-16550 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.1.1.0/24 Virtual Network Connection virtualHubs/hub1/hubVirtualNetworkConnections/spoke11 virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
10.1.2.0/24 Virtual Network Connection virtualHubs/hub1/hubVirtualNetworkConnections/spoke12 virtualHubs/hub1/hubVirtualNetworkConnections/spoke12
0.0.0.0/0 65012 HubBgpConnection virtualHubs/hub1/bgpConnections/spoke12 virtualHubs/hub1/bgpConnections/spoke12
10.1.22.0/24 65012 HubBgpConnection virtualHubs/hub1/bgpConnections/spoke12 virtualHubs/hub1/bgpConnections/spoke12
10.1.21.0/24 65012 HubBgpConnection virtualHubs/hub1/bgpConnections/spoke12 virtualHubs/hub1/bgpConnections/spoke12
168.63.129.16/32 65012 HubBgpConnection virtualHubs/hub1/bgpConnections/spoke12 virtualHubs/hub1/bgpConnections/spoke12
10.2.1.0/24 65520-65520 Remote Hub virtualHubs/hub2 virtualHubs/hub2
10.2.2.0/24 65520-65520 Remote Hub virtualHubs/hub2 virtualHubs/hub2
10.2.21.0/24 65520-65520-65012 Remote Hub virtualHubs/hub2 virtualHubs/hub2
10.2.22.0/24 65520-65520-65012 Remote Hub virtualHubs/hub2 virtualHubs/hub2
VPN vs ExpressRoute
Alright, we have explored the routing preferences of ExpressRoute and ASPath, but what about the VpnGateway option? I simplified the lab topology by removing one of the two regions and the indirect spokes, and adding a VPN connection to the remaining region:
As you can see, the VPN router is advertising exactly the same prefix as the ExpressRoute router. By know you should already know who is going to win with the default configuration (Hub Routing Preference being ExpressRoute):
❯ az network vhub list -o table
AddressPrefix AllowBranchToBranchTraffic HubRoutingPreference Location Name PreferredRoutingGateway ProvisioningState ResourceGroup RoutingState VirtualRouterAsn
--------------- ---------------------------- ---------------------- ------------------ ------ ------------------------- ------------------- --------------- -------------- ------------------
192.168.0.0/23 False ExpressRoute germanywestcentral hub1 ExpressRoute Succeeded vwan Provisioned 65515
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix ASPath NextHopType NextHop Origin
--------------- ----------------- -------------------------- ------------------------------------------------------ ---------------------------------------------------------------------------------
20.113.1.136/32 VPN_S2S_Gateway vpnGateways/hubvpn1 vpnGateways/hubvpn1
10.11.11.11/32 65501 VPN_S2S_Gateway vpnGateways/hubvpn1 vpnGateways/hubvpn1
10.201.0.0/26 65501 VPN_S2S_Gateway vpnGateways/hubvpn1 vpnGateways/hubvpn1
10.201.0.0/24 65501 VPN_S2S_Gateway vpnGateways/hubvpn1 vpnGateways/hubvpn1
0.0.0.0/0 65501 VPN_S2S_Gateway vpnGateways/hubvpn1 vpnGateways/hubvpn1
10.225.0.0/24 12076-65001-16550 ExpressRouteGateway expressRouteGateways/hub1ergw expressRouteGateways/hub1ergw
10.1.1.0/24 Virtual Network Connection virtualHubs/hub1/hubVirtualNetworkConnections/spoke11 virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
What if you preferred sending traffic over VPN Gateway primarily instead? Easy: change the preference to VPN Gateway. After doing so, here is what it looks like:
❯ az network vhub list -o table
AddressPrefix AllowBranchToBranchTraffic HubRoutingPreference Location Name PreferredRoutingGateway ProvisioningState ResourceGroup RoutingState VirtualRouterAsn
--------------- ---------------------------- ---------------------- ------------------ ------ ------------------------- ------------------- --------------- -------------- ------------------
192.168.0.0/23 False VpnGateway germanywestcentral hub1 ExpressRoute Succeeded vwan Provisioned 65515
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix ASPath NextHopType NextHop Origin
--------------- ----------------- -------------------------- ------------------------------------------------------ ---------------------------------------------------------------------------------
20.113.1.136/32 VPN_S2S_Gateway vpnGateways/hubvpn1 vpnGateways/hubvpn1
10.201.0.0/24 65501 VPN_S2S_Gateway vpnGateways/hubvpn1 vpnGateways/hubvpn1
10.201.0.0/26 65501 VPN_S2S_Gateway vpnGateways/hubvpn1 vpnGateways/hubvpn1
10.11.11.11/32 65501 VPN_S2S_Gateway vpnGateways/hubvpn1 vpnGateways/hubvpn1
0.0.0.0/0 65501 VPN_S2S_Gateway vpnGateways/hubvpn1 vpnGateways/hubvpn1
10.225.0.0/24 65501 VPN_S2S_Gateway vpnGateways/hubvpn1 vpnGateways/hubvpn1
10.1.1.0/24 Virtual Network Connection virtualHubs/hub1/hubVirtualNetworkConnections/spoke11 virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
If you want to send certain prefixes via the VPN tunnel, but others via ExpressRoute, that is where you would pick the routing preference of ASPath as we saw earlier in the article.
All in all: go for ASPath
With this new knob in Azure Virtual WAN hubs you have now more flexibility to choose how Azure will pick its best path. In my opinion ASPath is the most easily understood approach, giving more predictable results and more flexibility. Nevertheless, it is good having the other two (the default ExpressRoute and VpnGateway) in case you need them.
Cloudtrooper 
@erjosito excellent write-up, thanks. Going to start testing this in our dev environment shortly, and this was a great synthesis. I was also wondering if you could point me to a past blog where you expounded on the relationship between static routes in a vWAN vHub vs. static routes created in a vWAN Virtual Network Connection. I think that you pointed out some inherent dependency between these two, but now can’t find it for the life of me! Thanks again for the stellar content here; you’ve uncovered many an obsure (but significant) architectural nuance and have helped my decision-making tremendously.
LikeLike
Hey @mikery, so happy they are useful! I put a diagram on when each route is used some time ago in the Azure docs, in the NVA scenario: https://docs.microsoft.com/azure/virtual-wan/scenario-route-through-nva. Is this what you mean?
LikeLike
Yes! Thank you! This specifically is what I’m referring to:
“However, in this scenario we need to think about which static routes to configure. Each static route will have two components, one part in the Virtual WAN hub telling the Virtual WAN components which connection to use for each spoke, and another one in that specific connection pointing to the concrete IP address assigned to the NVA (or to a load balancer in front of multiple NVAs)”
which suggests that there is a dependency between these two types of static routes. I’ve noticed an issue when deploying specific VNET connections without any defined routes (using bicep) – it ~removes~ the next hop IP from the routing config in the Virtual WAN hub. I suspected it was a bug, but what you describe above means that these two route settings work in concert with one another, i.e.
– the vWAN Hub route simply places the traffic on the correct connection,
– the selected connection then knows the precise next hop in which to send it?
is that correct?
LikeLike
That’s correct. The UI simplifies the configuration of the route by allowing configuring both routes (at the rt and the cx level) at the same time, which can create confusion. The next hop of the RT’s route is the connection ID, and the next hop of the connection’s route is the IP
LikeLiked by 1 person