You might have set your Azure Vnet, with some NSGs associated to the subnets. You start rolling apps, to the point where you have many VMs, and many NSGs. Somebody makes an application upgrade, or installs a new application, but traffic is not flowing through. Which NSG is dropping traffic? Which TCP ports should be opened?
One possibility is using Traffic Analytics, a component of Azure Network Watcher, to get an idea of the traffic you have in your vnet. Traffic Analytics is a two-step process:
- NSG logs are stored in a storage account
- NSG logs from the storage account are processed (consolidated and enriched with additional information) and made queriable
You can find more information about Azure Traffic Analytics here.
One of the nicest features of Traffic Analytics in particular (and Azure Log Analytics in general) is being able to query logs with the KQL (Kusto Query Language). For example, you can use this query to find out the dropped flows in the last 3 hours for IP address 18.104.22.168:
AzureNetworkAnalytics_CL | where TimeGenerated >= ago(1h) | where SubType_s == "FlowLog" | where DeniedInFlows_d > 0 or DeniedOutFlows_d > 0 | where SrcIP_s == "22.214.171.124" | project NSGName=split(NSGList_s, "/"),NSGRules_s,DeniedInFlows_d,DeniedOutFlows_d,SrcIP_s,DestIP_s,DestPort_d,L7Protocol_s
You can find a reference guide for the Kusto Query Language here.
However, you will notice that there is a time lag, and you will not find the very latest logs in Log Analytics. The original NSG Flow logs are stored in storage account, in JSON format, so an option is getting those logs using the Azure Storage SDK.
That is exactly what the Python script in this Github repository does (assuming you have the Python SDK for storage installed). You can use different flags, like the –help option to get usage information:
$ python3 ./get_nsg_logs.py --help usage: get_nsg_logs.py [-h] [--accountName ACCOUNTNAME] [--displayLB] [--displayAllowed] [--displayDirection DISPLAYDIRECTION] [--displayHours DISPLAYHOURS] [--verbose] Get the latest flow logs in a storage account optional arguments: -h, --help show this help message and exit --accountName ACCOUNTNAME you need to supply an storage account name. You can get a list of your storage accounts with this command: az storage account list -o table --displayLB display or hide flows generated by the Azure LB (default: False) --displayAllowed display as well flows allowed by NSGs (default: False) --displayDirection DISPLAYDIRECTION display flows only in a specific direction. Can be in, out, or both (default in) --displayHours DISPLAYHOURS How many hours to look back (default: 1) --verbose run in verbose mode (default: False)
There is something you need to do before being able to access Azure Blob Storage: finding out the Azure Storage Account key. The script will read it from the environment variable STORAGE_ACCOUNT_KEY. There are multiple ways of putting the storage account key into an environment variable, I personally like this command:
export STORAGE_ACCOUNT_KEY=$(az storage account keys list -n your_storage_account_name --query .value -o tsv)
Once you have your environment variable set, you can start using the script. For example, in order to show dropped and allowed traffic (flag –displayAllowed) of ingress NSG logs (ingress is the default for the flag –displayDirection) stored in the storage account ernetworkhubdiag857, excluding Azure LB probe traffic (that is the default for the flag –displayLB) for the last 6 hours:
$ python3 ./get_nsg_logs.py --accountName ernetworkhubdiag857 --displayHours 6 --displayDirection in --displayAllowed 2018-09-21T09:49:57.3055150Z NVA-NSG DefaultRule_AllowVnetInBound A I 10.90.15.47 29014 10.139.149.70 23 2018-09-21T09:49:57.3055150Z NVA-NSG DefaultRule_AllowVnetInBound A I 10.90.15.47 29014 10.139.149.70 23 2018-09-21T09:49:57.3055150Z NVA-NSG DefaultRule_AllowVnetInBound A I 10.90.15.47 32069 10.139.149.70 23 2018-09-21T09:49:57.3055150Z NVA-NSG DefaultRule_AllowVnetInBound A I 10.90.15.47 29014 10.139.149.70 23 2018-09-21T09:49:57.3055150Z NVA-NSG DefaultRule_AllowVnetInBound A I 10.90.15.47 29014 10.139.149.70 23 2018-09-21T09:49:57.3055150Z NVA-NSG DefaultRule_AllowVnetInBound A I 10.90.15.47 32069 10.139.149.70 23