Get certificates with Azure Key Vault extension to your Linux VMs

Certificate management is one of those IT disciplines that is nobody’s dream, and still it can have quite a dramatic (negative) impact in your web presence if not done properly, such as users being told by the browser that your site is not secure. Azure has a nice little tool to manage certificates and bring them to your virtual machines, but it is not that well documented: welcome to the Azure Key Vault extension.

Prompted by my awesome colleague Bruna Moreira, I decided to have a look at it. Long story short: it does what it promises (copying and refreshing digital certificates from Azure Key Vault into your virtual machine) and it does it well, but you need to know a couple of little secrets to make it work.

By the way, if you are one of those persons that prefer reading code than plain English, you might want to go straight to my little Azure CLI script that contains the end solution.

Is the AKV extension your only option?

No, but in my opinion it is the best. Here other techniques you could try:

• Of course, you could reinvent the wheel yourself: if you install Azure CLI or the Azure Powershell module in your virtual machine, you can use a crontab script to periodically download the certificate from Azure Key Vault. But you would need to manage quite a number of moving parts, such as your script, the versioning of Azure CLI / Powershell module, etc.
• You have the option to copy secrets at VM creation time, for example with the Azure CLI command az vm create and the parameter --secrets. However, this will not refresh the certificate when it is renewed. To my surprise, this is the method that this tutorial to deploy nginx in an Azure VM is recommending (I will try to update that doc).
• You can of course download to your local computer and SCP it from there to your Azure virtual machine. Even if this might be the easiest for a lab, I don’t think you would want to do this in production.

Official docs

The Azure Key Vault extension has two variants: one for Windows, and the one we are going to look at in this post for Linux. Watch out for the Linux flavor support, it is not that broad at this point in time. I have tested it in Ubuntu 22.04.

If we have a look at the deployment options, it offers a description to create the extension with ARM, with PowerShell and with Azure CLI. The parameters are the same for all deployment mechanisms, so I will just pick one (if you have read some of my previous posts, you already know that I will choose Azure CLI). The command recommended by the docs is the following:

az vm extension set -n "KeyVaultForLinux" `
     --publisher Microsoft.Azure.KeyVault `
     -g "" `
     --vm-name "" `
     --version 2.0 `
     --enable-auto-upgrade true `
     --settings '{\"secretsManagementSettings\": { \"pollingIntervalInS\": \"\", \"certificateStoreName\": \"\", \"certificateStoreLocation\": \"\", \"observedCertificates\": [\"  \", \"  \"] }}'

You might have questions about this command, which I will try to answer in the next section of the post. And yes, before you ask, I will try to update the official docs too. Writing a blog post is much quicker though.

• Firstly, you don’t need the parameter certificateStoreName at all, since that parameter is only relevant for the Windows extension. You can safely remove it from the command.
• The observedCert1, observedCert2, etc. need to be specified in the format that references the secret where Key Vault certificates are stored. You might be confused here, since certificates and secrets are two different things in Key Vault parlance (see AKV secrets, certificates and keys). However, all certificates have a reference to an internal secret, which you can get for example with this Azure CLI command:

az keyvault certificate show \
    -n $cert_name --vault-name $akv_name \
    --query sid -o tsv

• The single quotes in the --settings parameter are probably a typo, and double quotes were meant.

• For the certStorageLocation you can specify a folder where the extension will place your certificate.

Armed with this knowledge, this is the command I use to deploy the extension to my virtual machine:

# Get the Secret ID corresponding to our certificate
cert_id=$(az keyvault certificate show --vault-name $akv_name --name $cert_name --query sid -o tsv)
# Construct a string with the settings for the AKV extension
akv_settings="{\"secretsManagementSettings\": { \"pollingIntervalInS\": \"3600\", \"certificateStoreLocation\": \"/etc/nginx/ssl\", \"observedCertificates\": [\"$cert_id\"] }}"
# Deploy AKV extension to VM
az vm extension set -g $rg --vm-name $vm_name -n "KeyVaultForLinux" --publisher Microsoft.Azure.KeyVault --version 2.0 --enable-auto-upgrade true --settings $akv_settings -o none

What happens now?

Alright, here is when it becomes interesting, because this is totally missing from the docs. What the extension will do is copying the certificate and the key in a single file with the format <akv_name>-<cert_name>, in the folder specified by the settings when you deployed the extension.

I have to admit that the previous paragraph is a bit of an oversimplification, because a closer look into the folder will show us that there is the actual file with the certificate, including its thumbnail in the name for easy identification (akv-vm.nginxcert.b876d86056d1410f869bc126e9062ca4.1694784287.1726407287.PEM), as well as a soft link to it that does not change with new versions of the certificate (akv-vm.nginxcert):

root@vm-nginx4:/etc/nginx/ssl# ls -al
total 24
drwx------ 2 root     root     4096 Sep 16 14:18 .
drwxr-xr-x 9 www-data www-data 4096 Sep 16 14:18 ..
lrwxrwxrwx 1 root     root       90 Sep 16 14:17 akv-vm.nginxcert -> /etc/nginx/ssl/akv-vm.nginxcert.b876d86056d1410f869bc126e9062ca4.1694784287.1726407287.PEM
-rw------- 1 root     root     2896 Sep 16 14:17 akv-vm.nginxcert.b876d86056d1410f869bc126e9062ca4.1694784287.1726407287.PEM

Let’s confirm that the downloaded file has both the cert and the key (stored as MD5 plain text, so you can use text tools such as cat or grep to inspect the file):

root@vm-nginx4:/etc/nginx/ssl# grep BEGIN ./akv-vm.nginxcert
-----BEGIN PRIVATE KEY-----
-----BEGIN CERTIFICATE-----

The first thing you might be thinking is that you don’t like having both the cert and the key in the same file, and I would agree with you. For example, nginx needs the certificate and the key in separate files. The extension will not do split the file for you, so you will need some openssl voodoo to create those distinct files:

# Split the file downloaded by the AKV extension in two (cert and key)
echo "Creating .key file with private key..."
openssl rsa -outform pem -in /etc/nginx/ssl/${akv_name}.${cert_name} -out /etc/nginx/ssl/${cert_name}.key
echo "Creating .crt file with certificate..."
openssl x509 -outform pem -in /etc/nginx/ssl/${akv_name}.${cert_name} -out /etc/nginx/ssl/${cert_name}.crt

You could run those two openssl commands right after the extension, but I would suggest to put them on a crontab entry running as frequently as the AKV extension polling period, so that if the extension downloads a new version of the certificate, you will split it again in cert and key. After running the script, we now see our two additional files in the folder:

root@vm-nginx4:/etc/nginx/ssl# ls -al
total 24
drwx------ 2 root     root     4096 Sep 16 14:18 .
drwxr-xr-x 9 www-data www-data 4096 Sep 16 14:18 ..
lrwxrwxrwx 1 root     root       90 Sep 16 14:17 akv-vm.nginxcert -> /etc/nginx/ssl/akv-vm.nginxcert.b876d86056d1410f869bc126e9062ca4.1694784287.1726407287.PEM
-rw------- 1 root     root     2896 Sep 16 14:17 akv-vm.nginxcert.b876d86056d1410f869bc126e9062ca4.1694784287.1726407287.PEM
-rw-r--r-- 1 root     root     1192 Sep 18 07:51 nginxcert.crt
-rw------- 1 root     root     1704 Sep 18 07:51 nginxcert.key

By the way, note that the folder where you copy the certificates needs to exist in advance, so you may want to make sure of that in your cloudinit file, which is a way to customize your virtual machine after deployment. Here is what my cloudinit file looks like to install nginx, configure a basic nginx web site with TLS, wait for the certificates to be copied by the extension, and split the certificate into the cert and the key parts (these last two actions are grouped in a script called convert_akv_cert.sh):

cat < $cloudinit_file
#cloud-config
package_upgrade: true
packages:
  - nginx
write_files:
  - owner: www-data:www-data
    path: /etc/nginx/sites-available/secure-server
    content: |
      server {
        listen 443 ssl http2;
        ssl_certificate /etc/nginx/ssl/$cert_name.crt;
        ssl_certificate_key /etc/nginx/ssl/$cert_name.key;
      }
      server {
            listen 80;
      }
  - owner: root:root
    path: /root/convert_akv_cert.sh
    permissions: "0755"
    content: |
        #!/bin/bash
        # Ideally this should be run as crontab entry, with the same frequency as the polling of the AKV extension
        # Wait until the AKV extension downloads the cert (a max counter to make sure this doesn't run forever would be nice)
        echo "Waiting for cert to be downloaded from AKV..."
        while [ ! -f /etc/nginx/ssl/${akv_name}.${cert_name} ] ; do
            sleep 5
        done
        # Split the file in two (cert and key)
        echo "Creating .key file with private key..."
        openssl rsa -outform pem -in /etc/nginx/ssl/${akv_name}.${cert_name} -out /etc/nginx/ssl/${cert_name}.key
        echo "Creating .crt file with certificate..."
        openssl x509 -outform pem -in /etc/nginx/ssl/${akv_name}.${cert_name} -out /etc/nginx/ssl/${cert_name}.crt
runcmd:
  - mkdir /etc/nginx/ssl
  - ln -s /etc/nginx/sites-available/secure-server /etc/nginx/sites-enabled/
  - rm /etc/nginx/sites-enabled/default
  - bash /root/convert_akv_cert.sh
  - (crontab -l 2>/dev/null; echo "0 * * * * /root/convert_akv_cert.sh") | crontab -
  - service nginx restart
EOF

Notice that the convert_akv_cert.sh script created by the cloudinit file might be executed before the Azure Key Vault extension has had time to run, and if that happens, it will not find any certificate to split in two files. In order to cope with that situation, the script will wait until the extension has downloaded a certificate with a simple while loop:

echo "Waiting for cert to be downloaded from AKV..."
while [ ! -f /etc/nginx/ssl/${akv_name}.${cert_name} ] ; do
    sleep 5
done

By the way, as a goodie you can see here how to add crontab entries from cloudinit, which is not so well documented out there.

How can I see that it is working?

I am glad you ask! I like quite a bit the logs generated by the extension. They are pretty self explanatory, and helped me to troubleshoot my initial mistakes (like not specifying the right secret ID, or trying to copy to a location that doesn’t exist).

These logs are stored in /var/log/azure/Microsoft.Azure.KeyVault.KeyVaultForLinux:

jose@vm-nginx4:/var/log/azure/Microsoft.Azure.KeyVault.KeyVaultForLinux$ ls -al
total 76
drwxr-xr-x 3 root root  4096 Sep 16 14:17 .
drwxr-xr-x 3 root root  4096 Sep 16 14:17 ..
-rw-r--r-- 1 root root  2643 Sep 16 14:17 CommandExecution.log
-rw-r--r-- 1 root root 57264 Sep 18 07:17 akvvm_service_2023-09-16_14-17-55.0.log
drwx------ 2 root root  4096 Sep 16 14:17 events

Here an extract of the logs file akvvm_service_2023-09-16_14-17-55.0.log, that shows the operations carried out:

[...]
2023-09-16 14:17:55:  [AuthClient]       AcquireTokenCallback invoked
2023-09-16 14:17:55:  [AuthClient]       acquiring token
2023-09-16 14:17:55:  [MSIAuthClient]    acquiring token via MSI
2023-09-16 14:17:55:  [MSIHttpClient]    MSI URL: http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&authority=https://login.microsoftonline.com/11111111-1111-1111-1111-111111111111&resource=https://vault.azure.net
[...]
2023-09-16 14:17:56:  [CertificateManager]        Installing latest version of 'https://akv-vm.vault.azure.net/secrets/nginxcert/b876d86056d1410f869bc126e9062ca4'.
2023-09-16 14:17:56:  [UnixCertificateStore]      certificate file name: '/etc/nginx/ssl/akv-vm.nginxcert.b876d86056d1410f869bc126e9062ca4.1694784287.1726407287.PEM'
2023-09-16 14:17:56:  [UnixCertificateStore]      certificate link name: '/etc/nginx/ssl/akv-vm.nginxcert'
2023-09-16 14:17:56:  [UnixCertificateStore]      No intermediate/root certificate exist or it's a self-signed certificate.
2023-09-16 14:17:56:  [CertificateManager]       Added ACL to certificate: https://akv-vm.vault.azure.net/secrets/nginxcert/b876d86056d1410f869bc126e9062ca4
[...]

You should see similar logs every hour (or whatever polling period you configured in the AKV extension settings).

Putting all together

Here you have the little script I wrote for Azure CLI with some diagnostics commands: https://github.com/erjosito/azcli/blob/master/akv_vm.azcli. Hopefully this post will save you some time the next time you need to deploy a digital certificate to an Azure virtual machine!