Azure SRE Agent for Networking

Posted on by erjosito

You might have heard about the Azure SRE Agent (https://aka.ms/sreagent). It is essentially an AI agent that reacts to alerts and troubleshoots the problems that generated them, and even remediates the root cause automatically if you will allow it. It is essentially an LLM with some custom prompts, skills and tools, specifically conceived for the Site Reliability Engineering (SRE) role. It can integrate with ticketing systems, monitoring tools and different environments that play a role in SRE.

There are multiple repos out there with starter kits and lab environments for the Azure SRE Agent, such as these ones:

There are probably more, but my issue with these is that they are mostly related to the application code. However, I often speak with infrastructure teams, so I decided to test the agent there. And to make sure that I am giving it hard tasks to solve, I chose the area I know best: networking.

Note that I didn’t say “Azure networking”, but just “networking”. I also tested it on a traditional network simulated with containerlab and FRR routers (although containerlab allows you to simulate all kind of network devices such as Arista, Cisco, Juniper, Nokia, VyOS, etc.), where I introduced an OSPF area mismatch issue to see how the SRE Agent reacted to it. Spoiler alert: it nailed it.

The repo

I put all the Infra-as-Code and the scripts in GitHub – erjosito/networking-sre-agent. There you can find the bicep to generate the environment and additional tooling such a script to introduce and fix a series of faults.

The Azure SRE Agent already comes with its own integration into the Microsoft Azure docs, but I enriched it with additional networking knowledge, which you can find in networking-sre-agent/knowledge at main. This is useful to make it aware of your topology, as well as to provide some guided troubleshooting strategies that you know to work. For example, I provided two runbooks to troubleshoot onprem BGP and OSPF issues. You should also provide it with some detailed information about your environment, so that it knows what it is looking at when an alert arrives, and it doesn’t need to waste time exploring the environment first. Additionally, giving it a “source of truth” that describes what the environment should look like (ideally your IaC repo for your Azure Landing Zone) will allow it to identify configuration drifts.

The repo creates a relatively complex environment where I can introduce all kind of sophisticated faults that I have encountered in my career, and that might have taken me too long to troubleshoot than I would care to admit. For example, I remember a customer situation where a route table was applied to the GatewaySubnet with gateway route propagation disabled, which broke the environment in subtle ways. That took me around 2 hours to figure out, and I have to admit that the SRE Agent beat me to it. But I am digressing, this is the Azure environment created by the bicep code in the repo:

                            ┌────────────────────────────────────┐
                            │          On-Prem VNet              │
                            │          10.100.0.0/16             │
                            │     VPN GW (BGP, ASN 65100)        │
                            │     Test VM (workload)             │
                            └───────────────┬────────────────────┘
                          S2S VPN (BGP)     │     S2S VPN (BGP)
                    ┌───────────────────────┴──────────────────────────┐
           ┌────────┴─────────────────────┐          ┌─────────────────┴────────────┐
           │        Hub 1 VNet            │   VNet   │        Hub 2 VNet            │
           │        10.1.0.0/16           │◄─Peering─►        10.2.0.0/16           │
           │                              │          │                              │
           │  NVA (Ubuntu+iptables)       │          │  NVA (Ubuntu+iptables)       │
           │  Internal LB (10.1.1.200)    │          │  Internal LB (10.2.1.200)    │
           │  VPN GW (BGP, ASN 65001)     │          │  VPN GW (BGP, ASN 65002)     │
           │  App Gateway + WAF           │          │  App Gateway + WAF           │
           │  Private Endpoint (10.1.4.4) │          │                              │
           │    └─► Storage Acct Static Web          │                              │
           └──┬──────────────────┬────────┘          └──┬──────────────────┬────────┘
        ┌─────┴─────┐      ┌─────┴─────┐          ┌─────┴─────┐      ┌─────┴─────┐
        │ Spoke 11  │      │ Spoke 12  │          │ Spoke 21  │      │ Spoke 22  │
        │ 10.11.    │      │ 10.12.    │          │ 10.21.    │      │ 10.22.    │
        │ 0.0/16    │      │ 0.0/16    │          │ 0.0/16    │      │ 0.0/16    │
        │ VM+Apache │      │ VM+Apache │          │ VM+Apache │      │ VM+Apache │
        └───────────┘      └───────────┘          └───────────┘      └───────────┘

               ┌────────────────────────────────────────────────────────────┐
               │              Traffic Manager (netsre-webapp)               │
               │          Endpoints: Hub1 AppGW PIP, Hub2 AppGW PIP         │
               └────────────────────────────────────────────────────────────┘

The repo also includes the scripts inject-fault.ps1 and inject-fault.sh, which can inject and remove a predefined set of faults. As you can see, some of these can be quite nasty:

> .\inject-fault.ps1 -List

Available fault injection scenarios:
====================================

  [IP Forwarding]
    ip-forwarding-hub1               Disable NIC-level IP forwarding on Hub1 NVA
    ip-forwarding-hub2               Disable NIC-level IP forwarding on Hub2 NVA
  [UDR]
    udr-wrong-nexthop                Set incorrect next-hop IP in spoke route table
    udr-missing-route                Remove the default route from spoke route table
    udr-detach                       Detach route table from spoke subnet
  [NSG]
    nsg-block-icmp                   Add high-priority NSG rule blocking ICMP
    nsg-block-all                    Add high-priority NSG rule blocking all traffic
    nsg-block-ssh                    Add high-priority NSG rule blocking SSH (port 22)
  [NVA]
    nva-iptables-drop                Drop all forwarded traffic via iptables on Hub1 NVA
    nva-iptables-block-spoke         Block traffic to/from a specific spoke via iptables
    nva-os-forwarding                Disable OS-level IP forwarding (sysctl) on Hub1 NVA
    nva-stop-ssh                     Stop the SSH service on Hub1 NVA
    nva-no-internet                  Block outbound internet traffic on NVA via iptables
  [VPN/BGP]
    vpn-disconnect                   Delete VPN connection between Hub1 and on-premises
    bgp-propagation                  Enable BGP route propagation on spoke route table (bypasses NVA)
    gw-disable-bgp-propagation       Disable BGP route propagation on gateway route table
    gateway-nsg                      Block VPN gateway traffic with NSG on GatewaySubnet
  [Peering]
    peering-disconnect               Remove VNet peering between Hub1 and Spoke11
    peering-no-gateway-transit       Disable gateway transit on hub-to-spoke peering
    peering-no-use-remote-gw         Disable 'use remote gateway' on spoke-to-hub peering
  [Private Link]
    pe-nsg-block                     Add NSG deny rule blocking traffic to the PE subnet (10.1.4.0/24)
    pe-dns-break                     Stop dnsmasq on Hub1 NVA, breaking PE DNS resolution from on-prem
    pe-route-missing                 Remove PE subnet UDR from spoke11 route table (traffic bypasses NVA)
    pe-dns-override                  Set spoke VNet DNS to Azure default and reboot VM — PE FQDN resolves to public IP while all other connectivity stays healthy
  [AppGW]
    appgw-probe-misconfigure         Set AppGW health probe host to 127.0.0.1 (backends become Unhealthy)
  [Containerlab]
    clab-ospf-area-mismatch          OSPF area mismatch on r1 transit (area 0->1) — adjacency drops, peer loopback withdrawn; because BGP peers over the loopbacks this tears down the BGP session and withdraws the LAN — clab CM FAILS + bgpd syslog
    clab-ospf-mtu-mismatch           OSPF interface MTU mismatch on r1 eth1 (1500->1400) — adjacency stuck in ExStart/Exchange, peer loopback never learned; BGP-over-loopback session drops — clab CM FAILS + bgpd syslog
    clab-ospf-network-type-mismatch  OSPF network-type mismatch on r1 eth1 (p2p->broadcast) — adjacency never reaches FULL, peer loopback not installed; BGP-over-loopback session drops — clab CM FAILS + bgpd syslog
    clab-bgp-session-down            Shut r1->r2 eBGP neighbor — LAN 172.31.20.0/24 withdrawn, clab CM fails
    clab-lan-route-withdraw          Remove r2 'network 172.31.20.0/24' from BGP — LAN route withdrawn while session stays Established, clab CM fails
    clab-bgp-prefix-filter           Apply r2 outbound route-map denying the LAN toward r1 — session Established but LAN not advertised, clab CM fails (subtle policy fault)
    clab-transit-link-down           Shut r1 eth1 transit — OSPF+BGP both down, LAN unreachable, clab CM fails
  [Combo]
    multi-fault                      Inject multiple faults simultaneously

Usage:
  .\inject-fault.ps1 -Scenario           # Inject fault
  .\inject-fault.ps1 -Scenario  -Revert  # Revert fault

The Azure issue

If you are too busy to deploy this, don’t worry, I have you covered. You can watch this 5-minute video (I removed the waiting times):

In this case I injected a User Defined Route (UDR) with the wrong next hop, also something not obvious, since you need to look at multiple screens to realize that the next hop of the UDR is not the same as the address of the Network Virtual Appliance (NVA).

It is strangely satisfying watching the agent explore the different possibilities and narrow down the root cause until it finds it and eventually remediates it.

The onprem issue

This was more challenging, because out of the bat the SRE Agent will have no information or knowledge to troubleshoot onprem devices. The first thing I did was bringing the onprem networking telemetry to Azure:

  • SNMP MIBs: I use a Telegraf SNMP collector that then exports the retrieved statistics to Azure Monitor as metrics.
  • Syslog messages: a standard syslog server with the Azure Monitoring Agent (AMA) to export the logs to a Log Analytics Workspace.
  • RADIUS accounting messages: similar structure to syslog, a FreeRADIUS server that exports the logs to Log Analytics via AMA.

The SRE Agent should also be able to run troubleshooting commands in the impacted network devices. In my case I had it use az vm run-command to run those commands from a jump box (where containerlab is running), but I could have used the new SRE Agent VNet integration. Maybe something for another time.

I also put on its knowledge base a couple of troubleshooting guides for BGP an OSPF focusing on my onprem environment. With that, I threw at it an OSPF area mismatch that made the router loopback interfaces unreachable and consequently made BGP also go down. Also here the Azure SRE Agent behave pretty well:

Conclusion

There are many things to optimize in this architecture before going to production, such as fine-tuning the knowledge base to eliminate unnecessary turnarounds, introducing incident deduplication, and configuring more aggressive alert generation intervals (the 5 minutes I used felt like an awfully long time). However, in my opinion it shows the great potential of Azure SRE Agent for areas that might not be in the main focus of your AI teams but can have a huge impact on your applications availability, such as networking.

I remember when I worked as third-level on-call support staff. When I was confronted with a problem, it always took me a good while to get familiar with it: finding and reading diagrams, running exploratory “show” commands, reading the previous troubleshooting tasks, etc. Having this kind of preliminary troubleshooting done automatically by the tooling would have been gold.

Have you tested Azure SRE Agent on infrastructure issues? Please let me know in the comments below!

Leave a comment