Thanks for the good feedback on this blog series! Here another set of questions I have been receiving lately: how does outbound connectivity look like for AKS pods? To answer that, we will look at how that works on both the Azure CNI and kubenet AKS clusters, deployed in one virtual network.

After deploying your Azure CNI and kubenet clusters and a test VM (see part 1 and part 2 of this blog series), let us have a look first at Azure CNI.

Other posts in this series:

  • Part 1: deep dive in AKS with Azure CNI in your own vnet
  • Part 2: deep dive in AKS with kubenet in your own vnet, and ingress controllers
  • Part 3 (this post): outbound connectivity from AKS pods
  • Part 4: NSGs with Azure CNI cluster
  • Part 5: Virtual Node
  • Part 6: Network Policy with Azure CNI

Outbound connections for pods in Azure CNI AKS clusters

We will start with the same whereami application that we used in the part 1 of this blog series (refer to it for more details):

k apply -f ./whereami.yaml

Let us check that the pods are deployed, and that the LoadBalancer service is provisioned:

$ k get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
aks-nodepool1-26711606-0 Ready agent 18m v1.11.5 10.13.76.4 <none> Ubuntu 16.04.5 LTS 4.15.0-1036-azure docker://3.0.1
$
$ k get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
whereami-564765b89-vdkjd 1/1 Running 0 11m 10.13.76.13 aks-nodepool1-26711606-0 <none>
whereami-564765b89-xcvjm 1/1 Running 0 11m 10.13.76.28 aks-nodepool1-26711606-0 <none>
$
$ k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 22m
whereami LoadBalancer 10.0.134.154 104.45.15.171 80:32188/TCP 11m

After your pods are deployed, we can start testing. First, let us look at connectivity inside of the vnet by pinging our test VM:

$ k exec whereami-564765b89-vdkjd -- ping 10.13.1.4
PING 10.13.1.4 (10.13.1.4) 56(84) bytes of data.
64 bytes from 10.13.1.4: icmp_seq=1 ttl=64 time=0.676 ms
64 bytes from 10.13.1.4: icmp_seq=2 ttl=64 time=1.49 ms

10.13.1.4 is the IP address of our test VM. If you tcpdump for ICMP packets while the previous ping is running, you will see the IP address the pod is coming from:

jose@testvm:~$ sudo tcpdump -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:11:58.729044 IP 10.13.76.13 > testvm: ICMP echo request, id 16, seq 33, length 64
22:11:58.729101 IP testvm > 10.13.76.13: ICMP echo reply, id 16, seq 33, length 64
22:11:59.753203 IP 10.13.76.13 > testvm: ICMP echo request, id 16, seq 34, length 64
22:11:59.753241 IP testvm > 10.13.76.13: ICMP echo reply, id 16, seq 34, length 64

As expected, the pod is coming from its own IP address, and not from the node’s IP.

What about our public IP address when going to the Internet? Let us check with the site ifconfig.co, that returns the IP address of the visitors:

$ k exec whereami-564765b89-vdkjd -- curl -s ifconfig.co
104.45.15.171

That IP address should look familiar, it is the one associated with the LoadBalancer service. This is interesting, because the pod it is coming from its own Azure IP configuration (where IP addresses for VMs) are defined, but still getting the LB’s IP address as source when browsing to the Internet

$ noderg=$(az aks show -g $rg -n $aksname_azure --query nodeResourceGroup -o tsv)
$ az network lb list -g $noderg --query [].[name,sku.name] -o tsv
kubernetes Basic

Something else to notice is that the Azure Load Balancer is of the Basic type. This is especially relevant for outgoing connections, since the basic Azure Load Balancer is going to allocate a limited amount of ephemeral ports per node (1,024, as described here). When AKS supports the standard Azure LB (in the meantime already supported on AKS-engine) this limit will disappear.

What about ingress controllers? Let us enable the HTTP routing addon of AKS, and deploy an app using an ingress:

$ az aks enable-addons -a http_application_routing -g $rg -n $aksname_azure
$ k apply -f ./whereami-ingress.yaml

If you look at all the services (since the nginx ingress controllers are deployed in kube-system), you will see two external IP addresses for LoadBalancer-type services:

$ k get svc --all-namespaces
NAMESPACE     NAME                                                  TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
default       kubernetes                                            ClusterIP      10.0.0.1       <none>          443/TCP                      41m
default       whereami                                              LoadBalancer   10.0.134.154   104.45.15.171   80:32188/TCP                 29m
default       whereami-ingress                                      ClusterIP      10.0.82.109    <none>          80/TCP                       33s
kube-system   addon-http-application-routing-default-http-backend   ClusterIP      10.0.209.122   <none>          80/TCP                       1m
kube-system   addon-http-application-routing-nginx-ingress          LoadBalancer   10.0.120.136   104.46.56.131   80:31474/TCP,443:30562/TCP   1m
kube-system   heapster                                              ClusterIP      10.0.157.125   <none>          80/TCP                       40m
kube-system   kube-dns                                              ClusterIP      10.0.0.10      <none>          53/UDP,53/TCP                40m
kube-system   kubernetes-dashboard                                  ClusterIP      10.0.42.100    <none>          80/TCP                       40m
kube-system   metrics-server                                        ClusterIP      10.0.28.34     <none>          443/TCP                      40m

And we have two additional pods:

$ k get po
NAME READY STATUS RESTARTS AGE
whereami-564765b89-vdkjd 1/1 Running 0 31m
whereami-564765b89-xcvjm 1/1 Running 0 31m
whereami-ingress-7f97d7c96f-6ws82 1/1 Running 0 2m
whereami-ingress-7f97d7c96f-qcf6p 1/1 Running 0 2m

Let’s see which IP address the new pods get:

$ k exec whereami-ingress-7f97d7c96f-qcf6p -- curl -s ifconfig.co
104.45.15.171

They get the public IP address associated to the first service! But that one does not have anything to do with the new pods. Why is that? Again, quoting Microsoft documentation: “When multiple public IP addresses are associated with Load Balancer Basic, any of these public IP addresses are a candidate for outbound flows, and one is selected at random“.

Outbound connectivity in Kubenet clusters

What about using kubenet? Let’s do a similar deployment as the one we just did for the Azure CNI. Refer to part 2 of this blog series for more details on how to deploy a kubenet AKS cluster in your own Vnet:

$ k config use-context kubenetcluster
$ k apply -f ./whereami.yaml
$ az aks enable-addons -a http_application_routing -g $rg -n $aksname_kubenet
$ k apply -f ./whereami-ingress.yaml

Let us gather the usual information:

$ k get node -o wide
NAME                       STATUS    ROLES     AGE       VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-nodepool1-35377101-0   Ready     agent     52m       v1.11.5   10.13.77.5    <none>        Ubuntu 16.04.5 LTS   4.15.0-1036-azure   docker://3.0.1
aks-nodepool1-35377101-1   Ready     agent     52m       v1.11.5   10.13.77.4    <none>        Ubuntu 16.04.5 LTS   4.15.0-1036-azure   docker://3.0.1
$
$ k get pod -o wide
NAME                                READY     STATUS    RESTARTS   AGE       IP             NODE                       NOMINATED NODE
whereami-564765b89-d8rdb            1/1       Running   0          4m        192.168.1.2    aks-nodepool1-35377101-1   <none>
whereami-564765b89-z7p82            1/1       Running   0          4m        192.168.0.9    aks-nodepool1-35377101-0   <none>
whereami-ingress-7f97d7c96f-2lqxf   1/1       Running   0          1m        192.168.0.10   aks-nodepool1-35377101-0   <none>
whereami-ingress-7f97d7c96f-66vvv   1/1       Running   0          1m        192.168.1.7    aks-nodepool1-35377101-1   <none>
$
$ k get svc --all-namespaces
NAMESPACE     NAME                                                  TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)                      AGE
default       kubernetes                                            ClusterIP      10.0.0.1       <none>            443/TCP                      56m
default       whereami                                              LoadBalancer   10.0.58.118    137.117.164.147   80:30694/TCP                 4m
default       whereami-ingress                                      ClusterIP      10.0.186.51    <none>            80/TCP                       1m
kube-system   addon-http-application-routing-default-http-backend   ClusterIP      10.0.254.32    <none>            80/TCP                       3m
kube-system   addon-http-application-routing-nginx-ingress          LoadBalancer   10.0.128.165   104.214.236.151   80:30058/TCP,443:30733/TCP   3m
kube-system   heapster                                              ClusterIP      10.0.165.163   <none>            80/TCP                       56m
kube-system   kube-dns                                              ClusterIP      10.0.0.10      <none>            53/UDP,53/TCP                56m
kube-system   kubernetes-dashboard                                  ClusterIP      10.0.80.140    <none>            80/TCP                       56m
kube-system   metrics-server                                        ClusterIP      10.0.45.29     <none>            443/TCP                      56m

Now, let’s try the private IP address of the VM in the same virtual network:

$ k exec whereami-564765b89-d8rdb -- ping 10.13.1.4
PING 10.13.1.4 (10.13.1.4) 56(84) bytes of data.
64 bytes from 10.13.1.4: icmp_seq=1 ttl=63 time=0.617 ms
64 bytes from 10.13.1.4: icmp_seq=2 ttl=63 time=1.75 ms
64 bytes from 10.13.1.4: icmp_seq=3 ttl=63 time=0.797 ms

And capture traffic at the VM:

jose@testvm:~$ sudo tcpdump icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:50:10.358307 IP 10.13.77.4 > testvm: ICMP echo request, id 11, seq 31, length 64
22:50:10.358320 IP testvm > 10.13.77.4: ICMP echo reply, id 11, seq 31, length 64

Interesting, traffic is sourced from 10.13.77.4, the IP of the node where our pod is running).

Let’s recheck intra-cluster communication, and do the same exercise from pod to pod. Let us start the ping from one pod to the other (note that the pods are in different nodes):

$ k exec whereami-564765b89-z7p82 -- ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=62 time=0.949 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=62 time=0.547 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=62 time=0.779 ms
^C

And now let’s see what we get at the other pod

$ k exec -it whereami-564765b89-d8rdb /bin/bash
[root@whereami-564765b89-d8rdb /]# yum install -y tcpdump
...
Complete!
[root@whereami-564765b89-d8rdb /]# tcpdump icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:00:24.579482 IP 192.168.0.9 > whereami-564765b89-d8rdb: ICMP echo request, id 12, seq 1, length 64
23:00:24.579511 IP whereami-564765b89-d8rdb > 192.168.0.9: ICMP echo reply, id 12, seq 1, length 64
23:00:25.599114 IP 192.168.0.9 > whereami-564765b89-d8rdb: ICMP echo request, id 12, seq 2, length 64
23:00:25.599138 IP whereami-564765b89-d8rdb > 192.168.0.9: ICMP echo reply, id 12, seq 2, length 64

As you can see, no SNAT is here at play, and the pods see each other’s IP addresses. How does that work? Let’s have a look at our friend iptables in one of the nodes (you might need to configure ssh public keys with the Azure CLI command `​az vm update`:

jose@aks-nodepool1-35377101-0:~$ sudo iptables -t nat -vxnL POSTROUTING
Chain POSTROUTING (policy ACCEPT 1 packets, 52 bytes)
pkts bytes target prot opt in out source destination
288142 18188250 KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
247724 15771644 MASQUERADE all -- * * 0.0.0.0/0 !192.168.0.0/16 /* kubenet: SNAT for outbound traffic from cluster */ ADDRTYPE match dst-type !LOCAL

Mmmh, the last entry is interesting. Translated to English it would be something like “Masquerade (SNAT) all traffic coming from everywhere going to outside the pod network”. This happens to be a flag you can pass to the local kubelet, called `non-masquerade-cidr`. kubelet runs in AKS as a process (not as a container), so we can have a look at it with the systemctl command (I use grep for better display in my shell, for some reason systemctl status without the grep was not breaking up the lines). I have reformatted the output in multiple lines for easier reading:

jose@aks-nodepool1-35377101-0:/etc/kubernetes$ systemctl status kubelet | grep /usr/local/bin/kubelet
└─3668 /usr/local/bin/kubelet --enable-server 
                              --node-labels=node-role.kubernetes.io/agent=,kubernetes.io/role=agent,agentpool=nodepool1,storageprofile=managed,storagetier=Premium_LRS,kubernetes.azure.com/cluster=MC_akstest_kubenetcluster_westeurope 
                              --v=2 
                              --volume-plugin-dir=/etc/kubernetes/volumeplugins 
                              --address=0.0.0.0 
                              --allow-privileged=true 
                              --anonymous-auth=false  
                              --authorization-mode=Webhook
                              --azure-container-registry-config=/etc/kubernetes/azure.json 
                              --cadvisor-port=0 --cgroups-per-qos=true
                              --client-ca-file=/etc/kubernetes/certs/ca.crt
                              --cloud-config=/etc/kubernetes/azure.json
                              --cloud-provider=azure
                              --cluster-dns=10.0.0.10
                              --cluster-domain=cluster.local
                              --enforce-node-allocatable=pods
                              --event-qps=0
                              --eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%
                              --feature-gates=PodPriority=true
                              --image-gc-high-threshold=85
                              --image-gc-low-threshold=80
                              --image-pull-progress-deadline=30m 
                              --keep-terminated-pod-volumes=false
                              --kube-reserved=cpu=69m,memory=1843Mi
                              --kubeconfig=/var/lib/kubelet/kubeconfig
                              --max-pods=110
                              --network-plugin=kubenet
                              --node-status-update-frequency=10s
                              --non-masquerade-cidr=192.168.0.0/16
                              --pod-infra-container-image=k8s.gcr.io/pause-amd64:3.1
                              --pod-manifest-path=/etc/kubernetes/manifests
                              --pod-max-pids=100

I was tempted to remove some lines from the previous update, but then I decided against it, since some of the options above are quite interesting (such as the max-pods limit or the kube reservations). But that is a story for another blog.

Finally, let us verify the IP addresses that are visible in the public Internet, but you probably know by now what is coming:

$ k exec whereami-564765b89-d8rdb -- curl -s ifconfig.co
137.117.164.147
$ k exec whereami-564765b89-z7p82 -- curl -s ifconfig.co
137.117.164.147
$ k exec whereami-ingress-7f97d7c96f-2lqxf -- curl -s ifconfig.co
137.117.164.147
$ k exec whereami-ingress-7f97d7c96f-66vvv -- curl -s ifconfig.co
137.117.164.147

As in the case of the Azure CNI cluster, the first public IP of the Azure Load Balancer is picked up for Internet communication to SNAT the pods.

And hereby we conclude this chapter of this blog series, I hope you learnt something today!