Private Link reality bites – service endpoints vs private link

Welcome to the sixth post in the Private Link Reality Bites series! Before we begin, let me recap the existing episodes of the series:

• Private Link reality bite #1: endpoints are an illusion (control plane vs data plane)
• Private Link reality bite #2: your routes might be lying (routing in virtual network gateways)
• Private Link reality bite #3: what’s my source IP (NAT)
• Private Link reality bite #4: Azure Firewall application rules
• Private Link reality bite #5: NXDomainRedirect (forwarding failed DNS requests to the Internet)
• Private Link reality bite #6: service endpoints vs private link

This question is as old as Private Link itself. Countless blog posts have already been written about it, not to mention this brief answer in the Private Link FAQ. Still, there are some interesting facts about the commonalities and differences between both technologies that haven’t been covered exhaustively elsewhere, so I will take a stab at this topic.

Let me summarize the story so far: first there were public IP addresses – this was the only way of accessing Azure services such as Azure Storage, Azure SQL, etc. Afterwards, many organizations asked for safer ways of connecting to those services, and Virtual Network Service Endpoints came along in August 2018. However, there were some missing pieces to that story, so around two years later in August 2020 Private Link was released for Azure Storage and Azure SQL Database. There are many different ways in which Azure PaaS services can take inbound connections and start outbound flows, see my post Taxonomy of PaaS access for a more detailed analysis.

Especially for customers that adopted VNet service endpoints between 2018 and 2020 there was a lot of confusion: should I migrate to Private Link or stay with Service Endpoints? What is the status of Service Endpoint Policies? Today VNet service endpoints are still available as a design option, and many organizations wonder whether they should go for one or the other.

Let me give you the short answer to that question, in case your time is limited: Private Link is the technology that you should probably look at. But let’s dive in.

Are private IP addresses more secure?

First of all, let’s debunk a common misconception. One of the features that Private Link gives you is the ability to access an Azure PaaS (Platform as a Service) offering via private IP addressing, both for source and destination. It is not rare hearing from certain organizations that they “need” Private Link because the public Internet is unsecure.

These individuals seem to be under the impression that if you use public IP addresses you automatically go over the public Internet and anybody can look at your data. Let me put an example to illustrate why this is not true: imagine if two persons, Scorch and Fixer, live in the same building in 1138 Azure Ave in Coruscant, and Scorch sends a letter in an envelope to Fixer:

Using public IP addresses is equivalent to writing the full address of both the sender and the recipient on the envelope. However, even if the addresses are globally routable, when Scorch gives the envelope to the receptionist of the building she is not going to put it in the public mail. She recognizes that Fixer lives in the same building, so she will just deliver the envelope directly to Fixer’s mailbox. This means that no public mail official will ever touch the email, even if the addresses are globally routable.

Going back to the matter at hand, the fact that you use public IP addresses for source and destination doesn’t mean that your traffic is going to be forwarded from Microsoft’s network to the public Internet and back. Instead, traffic between Azure virtual machines and Azure PaaS offerings will always stay in Microsoft’s backbone, even if it is going across regions and regardless how you set the routing preference of your public IPs and your PaaS resources.

If it is not keeping the traffic in Microsoft’s backbone, what benefits do technologies such as VNet service endpoints and private link bring to the table? It is essentially two:

• Imposing restrictions on who can access data.
• Reducing risk for data exfiltration.

Who can access my service?

Let’s start with the first one of those two. Initially, most PaaS (Platform as a Service) offerings such as Azure Storage only relied on authentication to make sure that only the right users got access to the data. But that is not enough, multi-layer security strategies demand to adopt network security measures on top of a solid authentication to prevent unwanted access.

However, when using public IP addresses it can be quite hard specifying that some virtual machines should be allowed to access data, but others shouldn’t. For example, imagine if you have a virtual network with two virtual machines VM1 and VM2:

If you want to have a consistent network inbound filter in your storage account, you need to be able to identify VM 1 with a predictable, unique public IP address, so that only access from that IP address will be allowed. Today you would do this either with NAT gateways (assuming that the virtual machines are in different subnets), but this technology wasn’t available in 2018. Back then, the most usual approach was assigning instance-level public IP addresses assigned to the VMs. This is a big “no-no” though, since instance-level public IPs mean opening up inbound flows from the Internet that you need to actively close with NSGs.

Enabling Service Endpoints

So Service Endpoints came to the rescue: they expose the private IP address of the Azure virtual machine that is trying to access a PaaS service, so that configuring inbound filters is easier and it doesn’t require adding public IP addresses to virtual machines.

The first thing you need to do is enabling service endpoints for the subnet where the Azure clients are located (note that only Azure clients are supported). For example, here is how you would enable the service endpoints for Azure Storage in a given subnet with Azure CLI:

az network vnet subnet update -n $subnet_name --vnet-name $vnet_name -g $rg --service-endpoints Microsoft.Storage.Global -o none

After enabling service endpoints for the subnet, if you inspect the effective routes of one of the virtual machines in that subnet you will notice that an interesting route has appeared:

This new route has a next hop of VirtualNetworkServiceEndpoint for 494 prefixes. What are those prefixes? If you check the one shown in the screenshot (191.239.224.0/26) in the list of Azure ranges, you will see that it is part of Azure Storage, more concretely in the East US 2 region. Inspecting the other 493 prefixes will show that all Azure Storage public prefixes in all Azure regions are covered. Consequently, this route essentially tells the Azure NIC to tunnel traffic from the virtual machine to any storage account’s public IP address so that the client’s private IP address is visible to it.

To verify that the private IP address is now visible to the PaaS resource (the storage account in our example) we can have a look at the access logs (focus on the CallerIpAddress column):

❯ az monitor log-analytics query -w $logws_customerid --analytics-query $blob_query -o table
AccountName               CallerIpAddress    StatusText            TableName      TimeGenerated
------------------------  -----------------  --------------------  -------------  ----------------------------
storagetest1138germany    10.13.76.36:34990  AuthorizationFailure  PrimaryResult  2025-02-12T14:06:59.8977648Z

So the client is still accessing the PaaS public IP address, but the storage account is now logging the private IP address of the caller virtual machine (which happens to be in a different region than the storage account, this is important because of what I explained in this previous post). You might be wondering, how is this possible? Addressing a service with its public IP address, but still keeping my private IP as source? The reason is some SDN voodoo happening under the covers, where traffic is encapsulated at the source (this is what that magic system route for all 494 Azure storage prefixes actually does).

In the post analogy it would be something like the following picture, where the destination address is globally routable, but the source address is not (note the data plane still stays inside of the building, meaning inside of Microsoft’s backbone):

Coming back to the access logs shown by the storage account, the status is an Authorization Failure. Why? Because service endpoints have two configuration components: first, you need to enable them in the source subnet (which we already did), and second, you need to configure the target PaaS to accept calls from that specific subnet, which looks like this in the Azure portal (the column ‘Endpoint status’ shows whether service endpoints have already been activated in the subnet as described above):

The previous screenshot is for Azure Storage accounts, most other services supporting virtual network service endpoints have similar network configurations where you enable subnets to access them.

Alright, mission accomplished, we are now able to very granularly (on a subnet basis) define who can access our PaaS services and who cannot, so that our data will only be accessed by certain virtual machines. Mission accomplished? Not quite, let’s have a look at our other goal: how to prevent folks to send data to unsanctioned data services from inside our network.

Preventing data exfiltration

This is the second part of the puzzle: even if your own PaaS resources are protected against unwanted access, how can you make sure that people in your organization are not uploading data to rogue storage accounts? Microsoft’s solution to this problem in the world of service endpoints were service endpoint policies, a technology that was only made available to storage accounts, but hasn’t been extended to other services or further developed since 2020.

The reason is that Private Link was introduced as a superset of service endpoint functionality to satisfy a requirement expressed by many customers who absolutely want to use private IP addresses to access their services, both for source and destination. Going back to our building analogy, essentially this:

Notice that the data plane of Private Link is actually very similar to using public IP addresses or service endpoints, and even the underlying technology to allow access from private IP addresses is essentially the same as for virtual network service endpoints (encapsulation at the source). However, this technology made security discussions much easier for organizations that didn’t want to see a single public IP address in their Azure footprint.

Another (and in some cases the main) reason why many organizations adopted Private Link endpoints as compared to VNet service endpoints is that it also enables access from your on-premises network, as shown in this post.

But back to our question: how do you prevent data exfiltration with the Private Link concept? Easy, you just deny access to all external public IP addresses in your firewall, since access to valid storage accounts will be East-West traffic using private IP addressing, potentially not even going through the firewall:

By having all involved IP addresses private, the firewall configuration is kept simple and throughput-intensive flows (such as storage-based) can go straight to the service without going through a firewall, which would increase cost and latency. Of course, if you really want to send private link traffic through the firewall you can, but you don’t have to.

Inbound rules with Private Link

Alright, we were able to define which subnets have access to our PaaS services with private endpoints. Can I do the same with Private Link? Yes! You can apply Network Security Groups to your private endpoint subnet (you need to enable the private endpoint policies for NSGs in the subnet though), and you can even associate Application Security Groups (ASGs) to the private endpoint to use it in other NSGs.

Note however that the security model with private endpoints is different from VNet service endpoints. With the latter you had to explicitly configure which subnets have access to the PaaS resource, otherwise no subnet had any access whatsoever. On the other hand, private endpoints per default allow access to all connections coming from your network, and if you want to only allow specific sources you need to explicitly configure Network Security Groups. So private links offer a fail-open security mechanism for internal IP addresses, while service endpoints implemented a fail-close concept.

Preventing data exfiltration with service endpoints

You can actually prevent data exfiltration to rogue PaaS resources without service endpoint policies (which are only available for Azure Storage but not for other resources), but it might be expensive. Let’s have a look a sample design:

Note that in this case the service endpoint would actually be configured in the firewall subnet, not in the virtual machines’ subnets (otherwise the route injected by the service endpoint would bypass the firewall). If all your traffic is anyhow going through the firewall (and this is the “expensive” part of it, specially for throughput-intensive applications such as storage), you can just define in your firewall which URLs to allow and which ones not to allow.

In the case of Azure Firewall I have tested using application rules for HTTPS traffic, I will leave it for another day testing what happens if the Azure Firewall sends the packets with the client’s original IP address as source (my guess is that it will not work under certain circumstances). Anyhow, the main drawback of this design is that the firewall would add cost and latency to the solution, especially if it is not bringing any value other than filtering based on the domain being accessed.

Conclusion

One of the main conclusions of this post is that whether you access Azure PaaS services using public IP addresses, virtual setwork service endpoints or private endpoints, the data plane doesn’t change much. However, by using private IP addresses for both the source and destination Private Link makes it easier to restrict access to your PaaS services and to avoid data exfiltration. Besides, private link supports access from on-premises systems, although you need to be careful with bandwidth-intensive operations.

Private Link allows either to send traffic to PaaS services directly from the clients, or to send it through a centralized firewall. Both options offer the possibility of filtering access to the PaaS resources (with NSGs) as well as preventing data exfiltration. This was not necessarily the case with VNet service endpoints, since for most data exfiltration scenarios you needed firewalls.

Do you prefer either private endpoints or service endpoints due to any reason that we haven’t looked at in this article (such as cost)? Please let me know in the comments!