CLI-based analysis of an ExpressRoute private peering

Quite frequently I see Azure connectivity diagrams that do not reflect accurately the topology of Azure Virtual Networks connnected to on-premises data centers via ExpressRoute. Additionally, I got the question last week of how to do some basic BGP troubleshooting in the involved networking devices in a way which is understandable by network administrators (read “with CLI”). Here we go!

The next picture shows how I like to draw ExpressRoute diagrams, essentially in a 3-tier fashion: the three involved components are the ExpressRoute Gateway (also known as Virtual Network Gateway, living inside of a Virtual Network), the Microsoft Edge Enterprise routers (located in an ExpressRoute location outside of the Azure region), and finally the customer edge routers. The “connection” Azure resources links an ExpressRoute Gateway with a pair of MSEEs (a circuit):

Each of these components will speak BGP to its immediate neigbors: the customer routers will peer with the MSEEs, and the MSEEs will speak BGP to the ExpressRoute gateways. The AS numbers of both the MSEEs and the gateways are not configurable: 12076 and 65515 respectively.

For reference, I deployed this test lab using this script, which leverages the Megaport APIs to configure the customer side of the ER connection. I am using a ZSH shell in my Windows Subsystem for Linux 2 running on Win11.

Looking at the MSEE

The Azure CLI offers some options to inspect routing at the MSEE level, using the network express-route command category. The only thing to notice is that you probably need to include the query value, since otherwise the parameter -o table cannot format the data correctly:

❯ az network express-route list-route-tables-summary -g $rg -n $er_circuit_name --path primary --peering-name AzurePrivatePeering --query value -o table
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Neighbor       V    AsProperty    UpDown    StatePfxRcd
-------------  ---  ------------  --------  -------------
169.254.85.41  4    65001         01:58:30  2
192.168.0.12   4    65515         01:46:31  1
192.168.0.13   4    65515         01:46:41  1

As you can see, the primary MSEE is peering with both ExpressRoute gateway instances (whose addresses are taken out of the GatewaySubnet in the VNet where they are deployed), and with a customer router (ASN 65001). You can issue a similar command for the secondary line:

❯ az network express-route list-route-tables-summary -g $rg -n $er_circuit_name --path secondary --peering-name AzurePrivatePeering --query value -o table
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Neighbor       V    AsProperty    UpDown    StatePfxRcd
-------------  ---  ------------  --------  -------------
169.254.85.45  4    65001         02:00:04  3
192.168.0.12   4    65515         01:48:33  1
192.168.0.13   4    65515         01:48:33  1

Once we have seen the BGP neighbors, we can have a look at the routes that the MSEE has learnt from them:

❯ az network express-route list-route-tables -g $rg -n $er_circuit_name --path primary --peering-name AzurePrivatePeering --query value -o table
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Network           NextHop        LocPrf    Weight    Path
----------------  -------------  --------  --------  -------
169.254.85.44/30  169.254.85.41            0         65001 ?
192.168.0.0/16    192.168.0.12             0         65515
192.168.0.0/16    192.168.0.13*            0         65515

You can see here that in the primary connection only one route is advertised from onprem (ASN 65001), and from Azure, the VNet prefix 192.168.0.0/16 is received from each gateway instance (192.168.0.12 and 192.168.0.13 in this example).

Looking at the ER Gateways

Let’s take a look first at the GatewaySubnet subnet in the VNet where the ER gateways have been deployed. As you can see here, it is configured with the prefix 192.168.0.0/24. Larger than required (a /27 would have been enough), but nothing wrong with it:

❯ az network vnet subnet list --vnet-name $vnet_name -g $rg -o table
AddressPrefix    Name           PrivateEndpointNetworkPolicies    PrivateLinkServiceNetworkPolicies    ProvisioningState    ResourceGroup
---------------  -------------  --------------------------------  -----------------------------------  -------------------  ---------------
192.168.1.0/24   vm             Enabled                           Enabled                              Succeeded            ertest
192.168.0.0/24   GatewaySubnet  Enabled                           Enabled                              Succeeded            ertest

If you remember (and if not feel free to scroll up), we had seen in the BGP neighbor list of the MSEE that the gateways had the IP addresses 192.168.0.12 and 192.168.0.13, which both fit inside of 192.168.0.0/24, the prefix for GatewaySubnet. When listing out the BGP neighbors for the ER gateway, the MSEEs sure appear there with their ASN 12076. Although interestingly enough, they appear with an IP address taken out of the GatewaySubnet:

❯ az network vnet-gateway list-bgp-peer-status -n $ergw_name -g $rg -o table
Neighbor     ASN    State      ConnectedDuration    RoutesReceived    MessagesSent    MessagesReceived
-----------  -----  ---------  -------------------  ----------------  --------------  ------------------
192.168.0.4  12076  Connected  00:24:53.0436193     2                 31              32
192.168.0.5  12076  Connected  00:24:42.9513303     3                 32              33

You might be wondering whether the previous command refers to each instance of the ExpressRoute gateway (we know there are two, .12 and .13). The command doesn’t show the local address, so we can try a customized version that will include the localAddress field, with a bit more sophisticated query parameter:

❯ az network vnet-gateway list-bgp-peer-status -n $ergw_name -g $rg --query 'value[].{LocalAddress:localAddress, Neighbor:neighbor, ASN:asn, State:state, Duration:connectedDuration, RoutesReceived:routesReceived}' -o table
LocalAddress    Neighbor     ASN    State      Duration          RoutesReceived
--------------  -----------  -----  ---------  ----------------  ----------------
192.168.0.12    192.168.0.4  12076  Connected  00:36:53.4728396  3
192.168.0.12    192.168.0.5  12076  Connected  00:36:53.8353536  2

Interesting, the list-bgp-peer-status command picked one of the gateway instances (the .12), and showed the neighbors for that one. I am not sure of how to get the neighbors of the other one, I guess we need to assume that they are identical, which is honestly a fair assumption to make, since both are peered to the MSEE as we saw. We can now have a look at the learnt routes:

❯ az network vnet-gateway list-learned-routes -n $ergw_name -g $rg -o table
Network           Origin    SourcePeer    AsPath       Weight    NextHop
----------------  --------  ------------  -----------  --------  ------------
192.168.0.0/16    Network   192.168.0.13               32768
169.254.85.40/30  EBgp      192.168.0.4   12076-65001  32779     192.168.0.4
169.254.85.40/30  EBgp      192.168.0.5   12076-65001  32779     192.168.0.5
169.254.85.44/30  EBgp      192.168.0.4   12076-65001  32779     192.168.0.4
169.254.85.44/30  EBgp      192.168.0.5   12076-65001  32779     192.168.0.5
192.168.0.0/16    EBgp      192.168.0.5   12076-12076  32779     192.168.0.12

Or if you prefer, with the customized version that includes the localAddress field (oddly enough, this command seems to pick the .13 instance, while the list-bgp-peer-status command took the .12 instance):

❯ az network vnet-gateway list-learned-routes -n $ergw_name -g $rg --query 'value[].{LocalAddress:localAddress, Peer:sourcePeer, Network:network, NextHop:nextHop, ASPath: asPath, Origin:origin, Weight:weight}' -o table
LocalAddress    Peer          Network           ASPath       Origin    Weight    NextHop
--------------  ------------  ----------------  -----------  --------  --------  ------------
192.168.0.13    192.168.0.13  192.168.0.0/16                 Network   32768
192.168.0.13    192.168.0.4   169.254.85.40/30  12076-65001  EBgp      32779     192.168.0.4
192.168.0.13    192.168.0.5   169.254.85.40/30  12076-65001  EBgp      32779     192.168.0.5
192.168.0.13    192.168.0.4   169.254.85.44/30  12076-65001  EBgp      32779     192.168.0.4
192.168.0.13    192.168.0.5   169.254.85.44/30  12076-65001  EBgp      32779     192.168.0.5
192.168.0.13    192.168.0.5   192.168.0.0/16    12076-12076  EBgp      32779     192.168.0.12

Here you can see a couple of things:

• The ER Gateway is getting the customer routes (169.254.85.40/30 and 169.254.85.44/30 in this example) from both MSEE routers, with the AS path 12076-65001
• The ER Gateways know the local VNet prefixes (marked as Origin Network)
• Each ER Gateway instance knows the local VNet prefix from the other instances advertised from the MSEE (the peer is 192.168.0.5). This is probably not too relevant in this scenario, but if we had other VNets connected to the circuit, they would all learn from each other via the MSEE

You can of course grep the previous command if you have many routes, and you are looking for specific ASNs, prefixes, or neighbors.

We can have a look at the advertised routes from the ER Gateway to one the MSEEs, for which we need to specify the peer we want to look at:

❯ az network vnet-gateway list-advertised-routes -n $ergw_name -g $rg --peer 192.168.0.4 --query 'value[].{LocalAddress:localAddress, Network:network, NextHop:nextHop, ASPath: asPath, Origin:origin}' -o table
LocalAddress    Network         NextHop       ASPath    Origin
--------------  --------------  ------------  --------  --------
192.168.0.13    192.168.0.0/16  192.168.0.13  65515     Igp

The output matches the learnt routes that we saw from the MSEE, where only the VNet prefix (192.168.0.0/16) was advertised from the ER gateways.

Customer side

Of course, these are your routers, so you should know how to inspect them. For completeness, since I am using a Megaport Cloud Router (MCR), here a screenshot of what the routes look like:

As you can see, the 192.168.0.0/16 is learnt from both MSEEs, and in this case the MCR is picking one of them (I guess the oldest route, if I remember the BGP selection criteria order), since the default for eBGP is not to perform ECMP.

It is interesting to note that the AS path is just 12076, and it doesn’t include the ER gateways ASN 65515.

That’s all!

Hopefully you learnt something from the main ideas in this post:

• Your routers connect via BGP to the MSEEs, the MSEEs connect via BGP to the ExpressRoute gateways
• The MSEEs are located in an ExpressRoute location, not in an Azure region.
• The ExpressRoute gateways are located in an Azure region (inside of a VNet)
• The ExpressRoute gateways have the ASN 65515, the MSEEs 12076 (both not configurable). The routes you will get from the MSEEs only contain 12076 as AS path

Please let me know in the comments if anything is not correct, or if you would like some deeper dive in any of these areas. Thanks for reading!