Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse @aiDotEngineer
Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse  @aiDotEngineer
Uploaded October 2026 | Updated November 2026, 4 days ago
A hands-on workshop on evaluating AI applications where mistakes are expensive. Annabell Schäfer and Doneyli De Jesus from Langfuse use a banking assistant to walk through the whole AI engineering loop. They cover tracing with PII masking, building datasets, turning domain-expert comments into failure modes, LLM-as-judge and code evaluators, and CI that blocks a prompt change when scores drop.

Chapters:
- 00:00 Why evals are hard
- 05:01 The AI engineering loop
- 12:47 Demo: a banking assistant with tracing and PII masking
- 20:35 Building datasets
- 25:18 Expert review and failure modes
- 30:07 Choosing evaluators
- 35:00 Running evals automatically on every prompt change
- 39:44 Dashboards for stakeholders
- 45:58 Working with your coding agent
- 52:13 Q&A

Doneyli De Jesus — langfuse.com
Annabell Schäfer — langfuse.com

Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, LangfuseAgents Made Code Free. Maintaining It Didnt Get Cheaper. - Dmitry Buykin, MaerskPlease… Enough with the Sloppy, One-Shot SaaS Replacements — Mahmoud Abdelwahab, RailwayHow OpenAI Built ChatGPT for Bankers and Analysts — Joseph Kim, OpenAIWithout Caching, Your AI Agent Costs 10x More - George Cameron, Artificial AnalysisYour AI Agent Is Hiding Its Mistakes From You - Ben Holmes, WarpYou Cant Tell Whats Real Anymore. Heres What Comes Next - Cristóbal Valenzuela, RunwayTeach Your Agent to Prove Its Done — Nick Nisi & Zack Proser, WorkOSScheduled Agents, Dreaming and Q&A: Claude Managed Agents (Part 3 of 4) — AnthropicWhen AI Agents Pay and Sellers Monetize: Building x402 Apps on AWS — Anil Nadiminti, AWSHow to Decide What Your AI Agent Should Remember - Wojciech Pluta, OracleHow Anthropic Builds: Lessons from Labs — Mike Krieger, Anthropic
AI Engineer |

Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER