Agents That Self Improve While You Sleep - Fuad Ali, Arize AI @aiDotEngineer
Agents That Self Improve While You Sleep - Fuad Ali, Arize AI  @aiDotEngineer
Uploaded October 2026 | Updated November 2026, 4 days ago
In this workshop, Fuad Ali of Arize AI shows how to build a self-improvement loop for AI agents: trace everything, let Signal (an always-on agent) read your traces on a schedule and file deduplicated issues, turn failures into evals and living regression datasets, run experiments to prove a fix works, and keep humans at the highest-leverage points: eval design, dataset curation and reviewing fixes. He demos the loop on a toy-shopping agent with a planted bug and takes questions on evaluating qualitative output, decision models and cost.

Chapters
00:00 Intro and workshop setup
05:42 Where is your team on the maturity curve?
07:39 The loop: trace, signal, eval, experiment, fix
08:46 Demo agent and tracing
12:15 Signal: an always-on agent reading your traces
16:11 Agent Studio and managed agents
18:31 Turning failures into datasets
20:32 Evals: code, LLM as a judge, decision models
24:48 Creating an eval from a Signal issue
27:01 Experiments: proving the fix works
31:12 Keeping humans in the loop
34:16 Toward a fully automated loop
40:16 Q&A: evaluating qualitative output
48:21 Q&A: decision models, data sources and cost

Fuad Ali, Arize AI

Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Agents That Self Improve While You Sleep - Fuad Ali, Arize AIThe Investors Edge Lives in Their Head — Anekha Sokhal, MoshiAI That Hunts Logic Bugs Before Attackers Do - Taylor Luttrell-Williams, SonarStop Making Your Customers Click Approve - Emil Chetty, WiseThinner Agents on a Smarter Substrate: The Ontology-based Semantic Layer — Emil Eifrem, Neo4jBetter AI Models Wont Fix Your Engineering Team — Nicholas Arcolano, JellyfishInsurance Claims Are AIs Messiest Data Problem — Denys Linkov, WisedocsHow Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, RampCan AI Understand a Distressed Credit? - Harneet Kaur, Axim IntelligenceThe One Thing Every AI Agent Framework Agrees On - Mislav Ivanda, DaytonaYour AI Isn’t the Problem. Your Infrastructure Is. — Jiachen Jiang, CoderWhy AI Startups Will Disappear Without Compute Markets — Evan Conrad, San Francisco Compute
AI Engineer |

Agents That Self Improve While You Sleep - Fuad Ali, Arize AI

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER