Uploaded August 2026 | Updated September 2026, 2 hours ago
Romulo Rocha (TandemTrace, BR)
Can a general-purpose LLM perform Tier-1 alert triage without expensive fine-tuning? Vendors promise "AI-driven SOCs," but measured evidence of where these systems actually succeed — and fall short — remains scarce.
This talk presents early findings from an ongoing controlled experiment evaluating LLM-based agents against synthetic alert datasets containing hidden threat campaigns and progressively adversarial noise. The evaluation examines how such agents reconstruct attack narratives, where standard retrieval approaches struggle, and how findings shift as model capability varies across rounds.
Attendees will leave with a practical framework for evaluating similar systems in their own environments, an honest view of the current limits of LLM-based triage, and a set of open questions that remain as the research continues.
---
With over 15 years of experience, Romulo Rocha is an Information Security leader specializing in building and scaling high-performance SOCs for mission-critical environments, including the Rio 2016 Olympic Games and Nubank. An expert in security automation, he focuses on evolving incident response beyond traditional SOAR using innovative architectures. His current research centers on evaluating Generative AI and autonomous agents within SOC pipelines to tackle alert fatigue and streamline complex triage. An active contributor to FIRST and the international security community, he is dedicated to sharing open, reproducible methodologies that advance the collective maturity of cyber defense.
Romulo Rocha (TandemTrace, BR)
Can a general-purpose LLM perform Tier-1 alert triage without expensive fine-tuning? Vendors promise "AI-driven SOCs," but measured evidence of where these systems actually succeed — and fall short — remains scarce.
This talk presents early findings from an ongoing controlled experiment evaluating LLM-based agents against synthetic alert datasets containing hidden threat campaigns and progressively adversarial noise. The evaluation examines how such agents reconstruct attack narratives, where standard retrieval approaches struggle, and how findings shift as model capability varies across rounds.
Attendees will leave with a practical framework for evaluating similar systems in their own environments, an honest view of the current limits of LLM-based triage, and a set of open questions that remain as the research continues.
---
With over 15 years of experience, Romulo Rocha is an Information Security leader specializing in building and scaling high-performance SOCs for mission-critical environments, including the Rio 2016 Olympic Games and Nubank. An expert in security automation, he focuses on evolving incident response beyond traditional SOAR using innovative architectures. His current research centers on evaluating Generative AI and autonomous agents within SOC pipelines to tackle alert fatigue and streamline complex triage. An active contributor to FIRST and the international security community, he is dedicated to sharing open, reproducible methodologies that advance the collective maturity of cyber defense.
