Back
#252
November 17, 2025

EP252 The Agentic SOC Reality: Governing AI Agents, Data Fidelity, and Measuring Success

Guests:

Topics:

Artificial Intelligence SIEM and SOC
29:29

Subscribe at Spotify

Subscribe at Apple Podcasts

Subscribe at YouTube

Topics covered:

  • Moving from traditional SIEM to an agentic SOC model, especially in a heavily regulated insurer, is a massive undertaking. What did the collaboration model with your vendor look like? 
  • Agentic AI introduces a new layer of risk - that of unconstrained or unintended autonomous action. In the context of Allianz, how did you establish the governance framework for the SOC alert triage agents?
  • Where did you draw the line between fully automated action and the mandatory "human-in-the-loop" for investigation or response?
  • Agentic triage is only as good as the data it analyzes. From your perspective, what were the biggest challenges - and wins - in ensuring the data fidelity, freshness, and completeness in your SIEM to fuel reliable agent decisions?
  • We've been talking about SOC automation for years, but this agentic wave feels different. As a deputy CISO, what was your primary, non-negotiable goal for the agent? Was it purely Mean Time to Respond (MTTR) reduction, or was the bigger strategic prize to fundamentally re-skill and uplevel your Tier 2/3 analysts by removing the low-value alert noise?
  • As you built this out, were there any surprises along the way that left you shaking your head or laughing at the unexpected AI behaviors?
  • We felt a major lack of proof - Anton kept asking for pudding - that any of the agentic SOC vendors we saw at RSA had actually achieved anything beyond hype! When it comes to your org, how are you measuring agent success?  What are the key metrics you are using right now?

Do you have something cool to share? Some questions? Let us know:

Transcript

This was a surprisingly grounded discussion featuring a heavily regulated entity (Allianz) moving faster on AI adoption than many Bay Area tech startups. The core theme was the transition from a traditional SIEM-based SOC to an "Agentic SOC." Unlike the "magic button" marketing we see at RSA, Alex and Lars presented a pragmatic maturity model, comparing the journey to autonomous driving levels. They argued that for a massive, global organization, AI agents aren't just cool tech—they are the only scalable way to handle the volume of modern threats (the "squelch" button concept).

Key takeaways include the treatment of AI agents as "employees" for identity governance, the absolute necessity of data quality before automation, and the massive efficiency gains (68 years of analyst time saved per quarter) achieved not by replacing humans, but by augmenting them.

Detailed Discussion Analysis

1. Defining the "Agentic SOC" (The Maturity Model)

Alex challenged the fluid definition of an "Autonomous SOC" by adopting a clear analogy based on self-driving cars:

Level 0: Manual operations (The "VW Bug").

Level 5: Fully autonomous (The "Knight Rider" or sitting reading a newspaper while the SOC runs itself).

Current State: Allianz places themselves at Level 2 moving into Level 3.

They emphasized that this is a journey, not a product purchase. The goal isn't to eliminate the human, but to create a "Co-pilot, not Autopilot" environment. This distinction is critical for regulatory compliance—humans must remain in the loop for destructive actions or significant decisions.

2. The Governance Paradox: Agents as Employees vs. Workloads

A fascinating debate emerged regarding how to govern these AI agents.

The "Employee" View: Allianz leans toward treating agents as "users" or "employees." They require an identity, they have access rights, and they operate within a hierarchy (e.g., an L1 Agent shouldn't have the rights to isolate the CEO's laptop, just as a human L1 analyst wouldn't).

The "Workload" View: Agents are just code/compute.

The Synthesis: Lars argued it’s a superposition. It is a workload (needs code security/infrastructure security) acting as an employee (needs Identity & Access Management).

3. Data Quality: The Silent Killer of AI Projects

Lars and Alex made a point that cannot be overstated: Garbage In, Garbage Out (GIGO). You cannot drop a sophisticated AI agent on top of a messy data swamp.

The Context Problem: A human analyst knows that an IP address in a log might be an internal range re-used from an external RFC range. An AI model doesn't "know" that context unless explicitly taught or grounded in that data.

The Fix: Before deploying agents, you must map data flows, parse correctly, and ensure the AI has access to business context (asset value, user roles).

Future State: They envision agents that don't just query a SIEM, but actively reach out to other systems (or even users via ChatOps) to pull missing context, mimicking a senior analyst's investigation flow.

4. Use Cases & "The Squelch Button"

The team introduced the concept of the "Squelch" knob (a radio term for noise suppression).

Traditional SOC: We tune detection rules down to avoid flooding analysts, effectively ignoring potential signals (false negatives) to manage workload.

Agentic SOC: You "turn up the noise." You ingest more alerts and let the AI agents handle the triage volume.

Malware Analysis: This was cited as a prime use case. Reverse engineering malware is high-skill, high-cost labor. Agents can now detonate, analyze, and map C2 (Command & Control) infrastructure faster than humans, often finding secondary C2 channels that standard sandboxes miss.

Risk Profile: Malware analysis is a "safe" place to fail. If the AI is wrong, a human re-checks. If an AI is wrong about blocking a business process, the cost is too high. Hence, deterministic automation is used for high-confidence actions (phishing blocks), while AI handles the "fuzzy" logic.

5. Metrics That Actually Matter

Instead of just measuring "Time to Detect" (which encourages bad behavior like closing tickets without looking), Allianz tracks:

Time Saved: They reported saving 68 years of analyst time per quarter. This is a staggering metric.

Agreement Rate: Running AI agents in parallel with human analysts to measure how often they agree.

Outcome Quality: Senior analysts spot-check AI decisions. Interestingly, they found cases where the AI was better than the human (lower false negative rate) because the AI didn't get "bored" or make assumptions that a signal was a false positive without checking.

Timeline of Key Topics

Introduction & The Singapore Origin Story: The hosts introduce the guests and reveal the episode was conceived at a karaoke session in Singapore (where Alex’s singing was apparently questionable).

Defining the Agentic SOC: Moving from the "VW Bug" (manual) to "Knight Rider" (autonomous). Allianz places themselves at Level 2/3.

Regulatory & Compliance Context: How a German insurer manages to innovate despite having ~120 different regulators globally.

AI as Co-pilot vs. Autopilot: The necessity of keeping a "human in the loop" for high-stakes decisions versus using AI for tedious tasks.

Governance & Identity: The philosophical and practical debate: Is an AI agent a workload or an employee? (Conclusion: Treat them like employees with IDs).

The "Superposition" of Agents: Combining deterministic workflows (for known threats) with probabilistic AI (for investigation).

Data Quality & "Garbage In, Garbage Out": The critical importance of clean data and business context (IP ranges, hostnames) before attempting AI.

Future Capabilities: Moving from "Triage" to "Investigation"—agents actively pulling data from systems rather than just reading logs.

The "Squelch" Analogy: Using AI to handle massive volumes of noise, allowing the SOC to lower detection thresholds and catch more subtle threats.

Metrics & ROI: Discussing the "68 years of time saved" metric and how to validate AI performance against human analysts (parallel runs).

Malware Analysis Deep Dive: Why this is the perfect use case—high labor cost, low risk of failure, and AI's ability to find multiple C2 channels.

Closing Advice: "Fix the Data First" and cultural tips (Will it make the boat go faster?).

View more episodes