Azure Virtual WAN Hub Routing Preference

You probably know Azure Virtual WAN: it is an Azure service that provides any-to-any connectivity across regions out of the box, or a “global transit network architecture”, as they describe here:

Virtual WAN Global Transit Architecture

Essentially Virtual WAN is a set of Microsoft-managed virtual hubs peered to each other, where you would connect your VNets and/or branches (ExpressRoute, Site-to-Site and Point-to-Site). One area that was a bit gray until now was about how to influence routing between hubs if there is more than one route, which is now configurable with the new and shiny Virtual Hub Routing Preference (in preview at the time of this writing).

In this blog I am going to dive deeper on how it works, and what effects does it have.

The test lab

I will start with this test bed: two virtual hubs, a spoke VNet peered to each virtual hub, and two ExpressRoute circuits each connected to both virtual hubs. This is a very common topology used by many customers:

Lab to test Hub Routing Preference

ExpressRoute routes are preferred by default

Let’s have a peak into hub1’s effective routes for the default route table (I am not using any custom routing, all connections are associated and propagating to the default route table):

❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix          ASPath             NextHopType                 NextHop                                                Origin
--------------  -----------------  --------------------------  ------------------------------------------------------ -----------------------------------------------------
10.225.0.0/24   12076-65001-16550  ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.1.1.0/24                        Virtual Network Connection  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
10.2.1.0/24     12076-12076        ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
192.168.2.0/23  12076-12076        ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.226.0.0/24   12076-65001-16550  ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw

By the way, don’t worry about the awk fanciness of the previous command, I just do it to make it a bit more readable.

If you have a look at the highlighted route for 10.2.1.0/24, you will see that the next hop is ExpressRoute. Why? Well, that is documented in the Virtual WAN FAQ and in that document describing the new Hub Routing Preference: in the presence of two routes with the same prefix and length, Virtual WAN will prefer the one coming from ExpressRoute. So what does the other route look like? We can have a look at the effective routes in the ExpressRoute connection:

❯ az network vhub get-effective-routes --resource-type ExpressRouteConnection --resource-id $hub1_ergw_cx0_id -g $rg -n hub1 -o table --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix          ASPath                         NextHopType                 NextHop                                                Origin
--------------  -----------------------------  --------------------------  ------------------------------------------------------ -----------------------------------------------------
10.225.0.0/24   12076-65001-16550              ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.225.0.0/24   65520-65520-12076-65001-16550  Remote Hub                  virtualHubs/hub2                                       virtualHubs/hub2
10.1.1.0/24                                    Virtual Network Connection  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
10.1.1.0/24     65520-65520-12076-12076        Remote Hub                  virtualHubs/hub2                                       virtualHubs/hub2
10.2.1.0/24     12076-12076                    ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.2.1.0/24     65520-65520                    Remote Hub                  virtualHubs/hub2                                       virtualHubs/hub2
192.168.2.0/23  12076-12076                    ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.226.0.0/24   12076-65001-16550              ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.226.0.0/24   65520-65520-12076-65001-16550  Remote Hub                  virtualHubs/hub2                                       virtualHubs/hub2

Interesting, here we do see both routes: one coming from ExpressRoute, where the ExpressRoute MSEE is reflecting the routes coming from hub2 back to hub1, and the other one directly from hub2. Let’s verify that hub1’s routing preference is set to ExpressRoute:

❯ az network vhub list -g $rg -o table
AddressPrefix    AllowBranchToBranchTraffic    HubRoutingPreference    Location            Name    PreferredRoutingGateway    ProvisioningState    ResourceGroup    RoutingState    VirtualRouterAsn
---------------  ----------------------------  ----------------------  ------------------  ------  -------------------------  -------------------  ---------------  --------------  ------------------
192.168.0.0/23   False                         ExpressRoute            germanywestcentral  hub1    ExpressRoute               Succeeded            vwaneastus       Provisioned     65515
192.168.2.0/23   False                         ExpressRoute            westcentralus       hub2    ExpressRoute               Succeeded            vwaneastus       Provisioned     65515

Note that this will be hard to find out with our old friend traceroute testing connectivity between the virtual machines in each region, since some of the components in Virtual WAN and ExpressRoute do not react friendly to TTLs expired in transit:

jose@spoke11-vm:~$ traceroute 10.2.1.4
traceroute to 10.2.1.4 (10.2.1.4), 64 hops max
  1   192.168.0.4  3.141ms  1.315ms  1.305ms
  2   *  *  *
  3   10.2.1.4  137.564ms  134.420ms  133.963ms

From this traceroute it is impossible telling if the next hop for our spoke VM is Virtual WAN or the ExpressRoute infrastructure.

Switching to ASPath preference

Changing the Hub Routing Preference to ASPath is pretty easy:

❯ routing_preference=ASPath  # Can be ASPath, ExpressRoute or VpnGateway
❯ az network vhub update -n hub1 -g $rg --hub-routing-preference $routing_preference -o none
❯ az network vhub update -n hub2 -g $rg --hub-routing-preference $routing_preference -o none
❯ az network vhub list -o table
AddressPrefix    AllowBranchToBranchTraffic    HubRoutingPreference    Location            Name    PreferredRoutingGateway    ProvisioningState    ResourceGroup    RoutingState    VirtualRouterAsn
---------------  ----------------------------  ----------------------  ------------------  ------  -------------------------  -------------------  ---------------  --------------  ------------------
192.168.2.0/23   False                         ASPath                  westcentralus       hub2    ExpressRoute               Succeeded            vwaneastus       Provisioned     65515
192.168.0.0/23   False                         ASPath                  germanywestcentral  hub1    ExpressRoute               Succeeded            vwaneastus       Provisioned     65515

Let’s check again the effective route tables in hub1’s default route table:

❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix          ASPath             NextHopType                 NextHop                                                Origin
--------------  -----------------  --------------------------  ------------------------------------------------------ ------------------------------------------------------
10.225.0.0/24   12076-65001-16550  ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.1.1.0/24                        Virtual Network Connection  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
192.168.2.0/23  12076-12076-12076  ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.226.0.0/24   12076-65001-16550  ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.2.1.0/24     65520-65520        Remote Hub                  virtualHubs/hub2                                       virtualHubs/hub2

There you go! Now Virtual WAN will send spoke-to-spoke traffic directly through the connection between the hubs, without hitting ExpressRoute.

BGP from the VNet

What about when you have an “indirect spoke” topology with BGP routes injected from a “direct spoke”? Let’s add those to our test lab:

Test lab with indirect spokes

With the Hub Routing Preference set to ExpressRoute here is what the default route table looks like:

❯ az network vhub list -o table
AddressPrefix    AllowBranchToBranchTraffic    HubRoutingPreference    Location            Name    PreferredRoutingGateway    ProvisioningState    ResourceGroup    RoutingState    VirtualRouterAsn
---------------  ----------------------------  ----------------------  ------------------  ------  -------------------------  -------------------  ---------------  --------------  ------------------
192.168.2.0/23   False                         ExpressRoute            westcentralus       hub2    ExpressRoute               Succeeded            vwaneastus       Provisioned     65515
192.168.0.0/23   False                         ExpressRoute            germanywestcentral  hub1    ExpressRoute               Succeeded            vwaneastus       Provisioned     65515

❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix            NextHopType                 NextHop                                                Origin                                                 ASPath
----------------  --------------------------  ------------------------------------------------------ ------------------------------------------------------ -----------------
10.225.0.0/24     ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw                          12076-65001-16550
192.168.2.0/23    ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw                          12076-12076
10.226.0.0/24     ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw                          12076-65001-16550
10.1.1.0/24       Virtual Network Connection  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
10.2.1.0/24       ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw                          12076-12076
10.1.2.0/24       Virtual Network Connection  virtualHubs/hub1/hubVirtualNetworkConnections/spoke12  virtualHubs/hub1/hubVirtualNetworkConnections/spoke12
10.2.2.0/24       ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw                          12076-12076
10.1.22.0/24      HubBgpConnection            virtualHubs/hub1/bgpConnections/spoke12                virtualHubs/hub1/bgpConnections/spoke12                65012
10.1.21.0/24      HubBgpConnection            virtualHubs/hub1/bgpConnections/spoke12                virtualHubs/hub1/bgpConnections/spoke12                65012
168.63.129.16/32  HubBgpConnection            virtualHubs/hub1/bgpConnections/spoke12                virtualHubs/hub1/bgpConnections/spoke12                65012
10.2.21.0/24      Remote Hub                  virtualHubs/hub2                                       virtualHubs/hub2                                       65520-65520-65012
10.2.22.0/24      Remote Hub                  virtualHubs/hub2                                       virtualHubs/hub2                                       65520-65520-65012

Now this is interesting: the remote “direct” spoke (10.2.2.0/24) is known both from ExpressRoute and the inter-hub link, so since our routing preference is ExpressRoute, we prefer that one. However, it looks like the remote “indirect” spokes are not known via ExpressRoute at all, so even with routing preference of ExpressRoute the inter-hub route is chosen (there is not any other one).

If you are wondering why ExpressRoute would only reflect routes coming from connections but not routes coming from BGP peerings, I am afraid I don’t have an answer for you.

And as we would expect, after switching the routing preference to ASPath all remote spokes (direct and indirect) go over the inter-hub link:

❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix            ASPath             NextHopType                 NextHop                                                                                                                                                          Origin
----------------  -----------------  --------------------------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------
10.225.0.0/24     12076-65001-16550  ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
192.168.2.0/23    12076-12076-12076  ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.226.0.0/24     12076-65001-16550  ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.1.1.0/24                          Virtual Network Connection  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11
10.1.2.0/24                          Virtual Network Connection  virtualHubs/hub1/hubVirtualNetworkConnections/spoke12  virtualHubs/hub1/hubVirtualNetworkConnections/spoke12
0.0.0.0/0         65012              HubBgpConnection            virtualHubs/hub1/bgpConnections/spoke12                virtualHubs/hub1/bgpConnections/spoke12
10.1.22.0/24      65012              HubBgpConnection            virtualHubs/hub1/bgpConnections/spoke12                virtualHubs/hub1/bgpConnections/spoke12
10.1.21.0/24      65012              HubBgpConnection            virtualHubs/hub1/bgpConnections/spoke12                virtualHubs/hub1/bgpConnections/spoke12
168.63.129.16/32  65012              HubBgpConnection            virtualHubs/hub1/bgpConnections/spoke12                virtualHubs/hub1/bgpConnections/spoke12
10.2.1.0/24       65520-65520        Remote Hub                  virtualHubs/hub2                                       virtualHubs/hub2
10.2.2.0/24       65520-65520        Remote Hub                  virtualHubs/hub2                                       virtualHubs/hub2
10.2.21.0/24      65520-65520-65012  Remote Hub                  virtualHubs/hub2                                       virtualHubs/hub2
10.2.22.0/24      65520-65520-65012  Remote Hub                  virtualHubs/hub2                                       virtualHubs/hub2

VPN vs ExpressRoute

Alright, we have explored the routing preferences of ExpressRoute and ASPath, but what about the VpnGateway option? I simplified the lab topology by removing one of the two regions and the indirect spokes, and adding a VPN connection to the remaining region:

Lab topology with ExpressRoute and VPN

As you can see, the VPN router is advertising exactly the same prefix as the ExpressRoute router. By know you should already know who is going to win with the default configuration (Hub Routing Preference being ExpressRoute):

❯ az network vhub list -o table
AddressPrefix    AllowBranchToBranchTraffic    HubRoutingPreference    Location            Name    PreferredRoutingGateway    ProvisioningState    ResourceGroup    RoutingState    VirtualRouterAsn
---------------  ----------------------------  ----------------------  ------------------  ------  -------------------------  -------------------  ---------------  --------------  ------------------
192.168.0.0/23   False                         ExpressRoute            germanywestcentral  hub1    ExpressRoute               Succeeded            vwan             Provisioned     65515
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix           ASPath             NextHopType                 NextHop                                                Origin
---------------  -----------------  --------------------------  ------------------------------------------------------ ---------------------------------------------------------------------------------
20.113.1.136/32                     VPN_S2S_Gateway             vpnGateways/hubvpn1                                    vpnGateways/hubvpn1
10.11.11.11/32   65501              VPN_S2S_Gateway             vpnGateways/hubvpn1                                    vpnGateways/hubvpn1
10.201.0.0/26    65501              VPN_S2S_Gateway             vpnGateways/hubvpn1                                    vpnGateways/hubvpn1
10.201.0.0/24    65501              VPN_S2S_Gateway             vpnGateways/hubvpn1                                    vpnGateways/hubvpn1
0.0.0.0/0        65501              VPN_S2S_Gateway             vpnGateways/hubvpn1                                    vpnGateways/hubvpn1
10.225.0.0/24    12076-65001-16550  ExpressRouteGateway         expressRouteGateways/hub1ergw                          expressRouteGateways/hub1ergw
10.1.1.0/24                         Virtual Network Connection  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11

What if you preferred sending traffic over VPN Gateway primarily instead? Easy: change the preference to VPN Gateway. After doing so, here is what it looks like:

❯ az network vhub list -o table
AddressPrefix    AllowBranchToBranchTraffic    HubRoutingPreference    Location            Name    PreferredRoutingGateway    ProvisioningState    ResourceGroup    RoutingState    VirtualRouterAsn
---------------  ----------------------------  ----------------------  ------------------  ------  -------------------------  -------------------  ---------------  --------------  ------------------
192.168.0.0/23   False                         VpnGateway              germanywestcentral  hub1    ExpressRoute               Succeeded            vwan             Provisioned     65515
❯ az network vhub get-effective-routes --resource-type RouteTable --resource-id $hub1_default_rt_id -g $rg -n hub1 --query 'value[].{Prefix:addressPrefixes[0],ASPath:asPath,NextHopType:nextHopType,NextHop:nextHops[0],Origin:routeOrigin}' -o table | awk '{ gsub(/\/subscriptions\/'$subscription_id'\/resourceGroups\/'$rg'\/providers\/Microsoft.Network\//,""); print }'
Prefix           ASPath             NextHopType                 NextHop                                                Origin
---------------  -----------------  --------------------------  ------------------------------------------------------ ---------------------------------------------------------------------------------
20.113.1.136/32            VPN_S2S_Gateway             vpnGateways/hubvpn1                                    vpnGateways/hubvpn1
10.201.0.0/24    65501     VPN_S2S_Gateway             vpnGateways/hubvpn1                                    vpnGateways/hubvpn1
10.201.0.0/26    65501     VPN_S2S_Gateway             vpnGateways/hubvpn1                                    vpnGateways/hubvpn1
10.11.11.11/32   65501     VPN_S2S_Gateway             vpnGateways/hubvpn1                                    vpnGateways/hubvpn1
0.0.0.0/0        65501     VPN_S2S_Gateway             vpnGateways/hubvpn1                                    vpnGateways/hubvpn1
10.225.0.0/24    65501     VPN_S2S_Gateway             vpnGateways/hubvpn1                                    vpnGateways/hubvpn1
10.1.1.0/24                Virtual Network Connection  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11  virtualHubs/hub1/hubVirtualNetworkConnections/spoke11

If you want to send certain prefixes via the VPN tunnel, but others via ExpressRoute, that is where you would pick the routing preference of ASPath as we saw earlier in the article.

All in all: go for ASPath

With this new knob in Azure Virtual WAN hubs you have now more flexibility to choose how Azure will pick its best path. In my opinion ASPath is the most easily understood approach, giving more predictable results and more flexibility. Nevertheless, it is good having the other two (the default ExpressRoute and VpnGateway) in case you need them.

4 thoughts on “Azure Virtual WAN Hub Routing Preference

  1. @erjosito excellent write-up, thanks. Going to start testing this in our dev environment shortly, and this was a great synthesis. I was also wondering if you could point me to a past blog where you expounded on the relationship between static routes in a vWAN vHub vs. static routes created in a vWAN Virtual Network Connection. I think that you pointed out some inherent dependency between these two, but now can’t find it for the life of me! Thanks again for the stellar content here; you’ve uncovered many an obsure (but significant) architectural nuance and have helped my decision-making tremendously.

    Like

    1. Hey @mikery, so happy they are useful! I put a diagram on when each route is used some time ago in the Azure docs, in the NVA scenario: https://docs.microsoft.com/azure/virtual-wan/scenario-route-through-nva. Is this what you mean?

      Like

      1. Mike Kery

        Yes! Thank you! This specifically is what I’m referring to:

        “However, in this scenario we need to think about which static routes to configure. Each static route will have two components, one part in the Virtual WAN hub telling the Virtual WAN components which connection to use for each spoke, and another one in that specific connection pointing to the concrete IP address assigned to the NVA (or to a load balancer in front of multiple NVAs)”

        which suggests that there is a dependency between these two types of static routes. I’ve noticed an issue when deploying specific VNET connections without any defined routes (using bicep) – it ~removes~ the next hop IP from the routing config in the Virtual WAN hub. I suspected it was a bug, but what you describe above means that these two route settings work in concert with one another, i.e.

        – the vWAN Hub route simply places the traffic on the correct connection,
        – the selected connection then knows the precise next hop in which to send it?

        is that correct?

        Like

      2. That’s correct. The UI simplifies the configuration of the route by allowing configuring both routes (at the rt and the cx level) at the same time, which can create confusion. The next hop of the RT’s route is the connection ID, and the next hop of the connection’s route is the IP

        Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s