If you are reading my blog you probably know what Virtual WAN and Azure Kubernetes Service are. You probably know as well that you can configure AKS so that egress traffic is sent through an Azure Firewall by using Azure routing as described in the article Control Egress Traffic in AKS. That article explains how to use the flag --outbound-type set to userDefinedRouting (which prevents a public Azure Load Balancer from being created for an AKS cluster), but how does this work with Virtual WAN?
There is a little dirty trick to make this work. People are still hitting this, so I decided to write a short blog post on this. TL;DR: you will need to trick AKS into thinking that it is not connected to Virtual WAN.
In essence, this is the traffic flow we are trying to implement:
If you configure an Virtual WAN hub as secured (including an Azure Firewall), and you configure it to secure Internet traffic, it will propagate a default route to the VMs (and AKS nodes are just VMs) in VNets attached. For example, in a sample VM I have in a VNet connected to a secured hub, these are the effective routes I see:
❯ az network nic show-effective-route-table -n testvmVMNic -g $rg -o table Source State Address Prefix Next Hop Type Next Hop IP --------------------- ------- ----------------- --------------------- ------------- Default Active 10.13.0.0/16 VnetLocal Default Active 192.168.0.0/23 VNetPeering VirtualNetworkGateway Active 0.0.0.0/0 VirtualNetworkGateway 192.168.0.132
The most important thing to note is that Virtual WAN injects routes to connected VNets without the need for UDRs. So let’s now create our AKS cluster and set the outbound type to userDefinedRouting! Unfortunately it won’t work, and this is the error we will get:
(ExistingRouteTableNotAssociatedWithSubnet) An existing route table has not been associated with subnet /subscriptions/blah-blah/resourceGroups/vwan/providers/Microsoft.Network/virtualNetworks/aksVnet/subnets/aks1stpool. Please update the route table association
Alright, not a problem! We can create a route table with gateway route propagation enabled (if you check again the routes injected by Virtual WAN in the first snippet, they appear as having the source VirtualNetworkGateway), attach it to the AKS node subnet, and we should be good! Unfortunately not, you would now get this other error message:
(RouteTableMissingDefaultRouteError) Default route 0.0.0.0/0 missing from route table /subscriptions/blah-blah/resourceGroups/vwan/providers/Microsoft.Network/routeTables/aks
Again a pretty explicit message! AKS really, really wants to see that 0.0.0.0/0 route there. OK, if they want it, we can give it to them! We can create a UDR route that will mimic the existing Virtual WAN route, and now the effective routes in our test VM would look like this:
❯ az network nic show-effective-route-table -n testvmVMNic -g $rg -o table Source State Address Prefix Next Hop Type Next Hop IP --------------------- ------- ----------------- --------------------- ------------- Default Active 10.13.0.0/16 VnetLocal Default Active 192.168.0.0/23 VNetPeering VirtualNetworkGateway Invalid 0.0.0.0/0 VirtualNetworkGateway 192.168.0.132 User Active 0.0.0.0/0 VirtualAppliance 192.168.0.132
As you can see we just created a UDR with exactly the same next hop IP address as the original default route injected by Virtual WAN. This route is completely useless, we only need it to fool the AKS creation process. Now we can successfully create our AKS cluster with outbound type set to userDefinedRouting so that no public Azure Load Balancer is created.
After you have created the cluster, you can remove the route or even the route-table, everything will work just fine (as far as I have tested). You can even create additional nodepools in different subnets, and AKS will not complain again.
This workaround should work equally well if instead of injecting routes through Virtual WAN you do it with Azure Route Server, since its behavior is very similar to Virtual WAN.
Please write it in the comments if you happen to find another situation where you need that dummy UDR to trick the AKS creation API!
Cloudtrooper 