Uploaded October 2026 | Updated November 2026, 4 days ago
In this workshop, Fuad Ali of Arize AI shows how to build a self-improvement loop for AI agents: trace everything, let Signal (an always-on agent) read your traces on a schedule and file deduplicated issues, turn failures into evals and living regression datasets, run experiments to prove a fix works, and keep humans at the highest-leverage points: eval design, dataset curation and reviewing fixes. He demos the loop on a toy-shopping agent with a planted bug and takes questions on evaluating qualitative output, decision models and cost.
Chapters
00:00 Intro and workshop setup
05:42 Where is your team on the maturity curve?
07:39 The loop: trace, signal, eval, experiment, fix
08:46 Demo agent and tracing
12:15 Signal: an always-on agent reading your traces
16:11 Agent Studio and managed agents
18:31 Turning failures into datasets
20:32 Evals: code, LLM as a judge, decision models
24:48 Creating an eval from a Signal issue
27:01 Experiments: proving the fix works
31:12 Keeping humans in the loop
34:16 Toward a fully automated loop
40:16 Q&A: evaluating qualitative output
48:21 Q&A: decision models, data sources and cost
Fuad Ali, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
In this workshop, Fuad Ali of Arize AI shows how to build a self-improvement loop for AI agents: trace everything, let Signal (an always-on agent) read your traces on a schedule and file deduplicated issues, turn failures into evals and living regression datasets, run experiments to prove a fix works, and keep humans at the highest-leverage points: eval design, dataset curation and reviewing fixes. He demos the loop on a toy-shopping agent with a planted bug and takes questions on evaluating qualitative output, decision models and cost.
Chapters
00:00 Intro and workshop setup
05:42 Where is your team on the maturity curve?
07:39 The loop: trace, signal, eval, experiment, fix
08:46 Demo agent and tracing
12:15 Signal: an always-on agent reading your traces
16:11 Agent Studio and managed agents
18:31 Turning failures into datasets
20:32 Evals: code, LLM as a judge, decision models
24:48 Creating an eval from a Signal issue
27:01 Experiments: proving the fix works
31:12 Keeping humans in the loop
34:16 Toward a fully automated loop
40:16 Q&A: evaluating qualitative output
48:21 Q&A: decision models, data sources and cost
Fuad Ali, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!










