Uploaded August 2026 | Updated September 2026, 2 weeks ago
Don't miss out! Join us at our next KubeCon + CloudNativeCon events in Shanghai, China (8-9 September, 2026) and Salt Lake City, United States (Nov 9–12, 2026). Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at kubecon.io
The Great Doubt: What Building an AI Agent Taught Us About Trust - Nicole van der Hoeven, Grafana Labs
When we started building an AI assistant for observability, we thought the hard part would be making it smart. We were wrong. The hard part was knowing when to trust it.
AI agents hallucinate, forget context, and confidently give wrong answers. Traditional testing doesn't catch these failures. What's needed is evaluation grounded in systematic doubt.
This talk shares lessons from shipping an AI agent to production: how to build a golden dataset of use cases you must get right, how to use LLM-as-judge when there's no ground truth, and how to use OpenTelemetry traces to debug eval failures. You'll also hear what's still unsolved: evaluating multi-agent handoffs and closing the feedback loop between what users ask and what your evals cover.
Kyoto School philosopher Nishitani Keiji called this "The Great Doubt" (大疑): questioning every assumption until only what survives is real. For AI agents, that's not philosophy. It's the job.
Don't miss out! Join us at our next KubeCon + CloudNativeCon events in Shanghai, China (8-9 September, 2026) and Salt Lake City, United States (Nov 9–12, 2026). Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at kubecon.io
The Great Doubt: What Building an AI Agent Taught Us About Trust - Nicole van der Hoeven, Grafana Labs
When we started building an AI assistant for observability, we thought the hard part would be making it smart. We were wrong. The hard part was knowing when to trust it.
AI agents hallucinate, forget context, and confidently give wrong answers. Traditional testing doesn't catch these failures. What's needed is evaluation grounded in systematic doubt.
This talk shares lessons from shipping an AI agent to production: how to build a golden dataset of use cases you must get right, how to use LLM-as-judge when there's no ground truth, and how to use OpenTelemetry traces to debug eval failures. You'll also hear what's still unsolved: evaluating multi-agent handoffs and closing the feedback loop between what users ask and what your evals cover.
Kyoto School philosopher Nishitani Keiji called this "The Great Doubt" (大疑): questioning every assumption until only what survives is real. For AI agents, that's not philosophy. It's the job.
