Uploaded July 2026 | Updated November 2026, 3 days ago
Learn how to turn the failures you've found into a regression dataset, then prove your fix actually works instead of just hoping it does. Finding bugs is only half the job.
A prompt tweak that "feels" better isn't proof. A controlled experiment against a saved regression set is what tells you the score change is real, not noise.
Watch this to learn:
β’ How to save failing spans as a dataset and use it as a regression test set
β’ How to run controlled experiments so score changes are attributable to your change, not chance
β’ Where to intervene: data, prompt, or model, and how eval-driven development flips the usual workflow
Chapters:
00:00 Using evals to improve the agent
00:36 Save failures as a dataset, your regression set
02:39 Let Claude propose the fix, grounded in explanations
03:55 Running a controlled experiment
04:48 Comparing results example by example
06:16 How many samples do you need?
06:29 Where to intervene: data, prompt, or model
07:20 Eval-driven development
08:22 Next: production and online evals
π Sign up for free: app.arize.com/auth/login?utm_source=youtube&utm_medium=organic_social&utm_campaign=arize_ax_getting_started
π Learn more about Arize AX: arize.com
π Colab notebook for the series: colab.research.google.com/drive/1dViThD0kJjbqtDGE7-ciBYIRwV-HBN1T
π Docs: docs.arize.com
π Subscribe for weekly content on LLMs, agents, and evaluation: youtube.com/@arizeai?sub_confirmation=1
#ArizeAX #AgentEvaluation #LLMEvaluation
Learn how to turn the failures you've found into a regression dataset, then prove your fix actually works instead of just hoping it does. Finding bugs is only half the job.
A prompt tweak that "feels" better isn't proof. A controlled experiment against a saved regression set is what tells you the score change is real, not noise.
Watch this to learn:
β’ How to save failing spans as a dataset and use it as a regression test set
β’ How to run controlled experiments so score changes are attributable to your change, not chance
β’ Where to intervene: data, prompt, or model, and how eval-driven development flips the usual workflow
Chapters:
00:00 Using evals to improve the agent
00:36 Save failures as a dataset, your regression set
02:39 Let Claude propose the fix, grounded in explanations
03:55 Running a controlled experiment
04:48 Comparing results example by example
06:16 How many samples do you need?
06:29 Where to intervene: data, prompt, or model
07:20 Eval-driven development
08:22 Next: production and online evals
π Sign up for free: app.arize.com/auth/login?utm_source=youtube&utm_medium=organic_social&utm_campaign=arize_ax_getting_started
π Learn more about Arize AX: arize.com
π Colab notebook for the series: colab.research.google.com/drive/1dViThD0kJjbqtDGE7-ciBYIRwV-HBN1T
π Docs: docs.arize.com
π Subscribe for weekly content on LLMs, agents, and evaluation: youtube.com/@arizeai?sub_confirmation=1
#ArizeAX #AgentEvaluation #LLMEvaluation
![How AI Ops Teams Will Manage Fleets of Agents | Arize
As AI teams deploy more agents, traces, and evals, the operational workload quickly becomes too large for humans to review manually.
In this conversation, Jim sits down with SallyAnn DeLucia, Head of Product at Arize, to discuss how AI operations teams are changing as agents become part of the team itself.
Instead of asking engineers and subject matter experts to sift through thousands of traces, teams can use specialized agents to investigate failures, find patterns, surface anomalies, evaluate cost, monitor safety, and identify where human judgment is actually needed.
The result is a different operating model: humans spend less time searching through telemetry and more time reviewing high-leverage decisions, providing domain expertise, and managing a growing fleet of specialized agents.
The conversation covers:
β’ Why AI observability and evals create a new operational bottleneck
β’ How agents can investigate traces and surface the highest-value issues
β’ Why humans should focus on review, annotation, and domain expertise
β’ Using agents to scale human annotation workflows
β’ Automating LLM eval alignment
β’ Agents for dataset coverage, eval quality, cost, safety, and security
β’ Why traces and evals remain essential even when agents consume the data
β’ How engineering roles change when agents handle more operational work
β’ Why systems thinking and domain expertise become more important
β’ What it looks like to manage a fleet of specialized AI agents
One example is eval alignment. An agent can monitor evaluator performance, identify when alignment work is needed, curate candidate examples, and hand the highest-value cases to a subject matter expert for review. The human still provides the judgment, but the agent removes much of the manual work around it.
SallyAnn also explains how this model could expand into fleets of agents responsible for different operational functions: cost optimization, security, dataset quality, evaluation health, classification, and production issue detection.
The key shift is that telemetry does not disappear. What changes is who consumes it.
Instead of humans manually combing through every trace and evaluation result, agents can process the data continuously and bring humans into the loop when their expertise has the highest impact. :contentReference[oaicite:1]{index=1}
Chapters:
00:00 The shift to agentic AI operations
02:29 What should AI agents handle?
04:20 Scaling the human annotation cycle
06:33 Automating eval alignment
08:13 Building a fleet of specialized agents
10:54 Why humans are still essential
12:48 The new engineering skill set
14:18 Advice for engineering managers
π Explore Arize AX: https://arize.com/ax
π Learn more about Arize: https://arize.com
π Subscribe for more videos on AI agents, evals, observability, and AI engineering:
https://www.youtube.com/@arizeai?sub_confirmation=1
#AIOps #AIAgents #AgentObservability How AI Ops Teams Will Manage Fleets of Agents | Arize](https://i.ytimg.com/vi/4zG3OPDGqac/mqdefault.jpg)









