How AI Ops Teams Will Manage Fleets of Agents | Arize @arizeai
How AI Ops Teams Will Manage Fleets of Agents | Arize  @arizeai
Uploaded September 2026 | Updated November 2026, 1 hour ago
As AI teams deploy more agents, traces, and evals, the operational workload quickly becomes too large for humans to review manually.

In this conversation, Jim sits down with SallyAnn DeLucia, Head of Product at Arize, to discuss how AI operations teams are changing as agents become part of the team itself.

Instead of asking engineers and subject matter experts to sift through thousands of traces, teams can use specialized agents to investigate failures, find patterns, surface anomalies, evaluate cost, monitor safety, and identify where human judgment is actually needed.

The result is a different operating model: humans spend less time searching through telemetry and more time reviewing high-leverage decisions, providing domain expertise, and managing a growing fleet of specialized agents.

The conversation covers:

• Why AI observability and evals create a new operational bottleneck
• How agents can investigate traces and surface the highest-value issues
• Why humans should focus on review, annotation, and domain expertise
• Using agents to scale human annotation workflows
• Automating LLM eval alignment
• Agents for dataset coverage, eval quality, cost, safety, and security
• Why traces and evals remain essential even when agents consume the data
• How engineering roles change when agents handle more operational work
• Why systems thinking and domain expertise become more important
• What it looks like to manage a fleet of specialized AI agents

One example is eval alignment. An agent can monitor evaluator performance, identify when alignment work is needed, curate candidate examples, and hand the highest-value cases to a subject matter expert for review. The human still provides the judgment, but the agent removes much of the manual work around it.

SallyAnn also explains how this model could expand into fleets of agents responsible for different operational functions: cost optimization, security, dataset quality, evaluation health, classification, and production issue detection.

The key shift is that telemetry does not disappear. What changes is who consumes it.

Instead of humans manually combing through every trace and evaluation result, agents can process the data continuously and bring humans into the loop when their expertise has the highest impact. :contentReference[oaicite:1]{index=1}

Chapters:
00:00 The shift to agentic AI operations
02:29 What should AI agents handle?
04:20 Scaling the human annotation cycle
06:33 Automating eval alignment
08:13 Building a fleet of specialized agents
10:54 Why humans are still essential
12:48 The new engineering skill set
14:18 Advice for engineering managers

🔗 Explore Arize AX: arize.com/ax
🔗 Learn more about Arize: arize.com
🔔 Subscribe for more videos on AI agents, evals, observability, and AI engineering:
youtube.com/@arizeai?sub_confirmation=1

#AIOps #AIAgents #AgentObservability
How AI Ops Teams Will Manage Fleets of Agents | ArizeArize AX: Alyx Demo 2026How a Simple To-Do List Fix Boosted Agent Completion RatesEveryone Is an AI Agent Builder (If You Let Them) | CrewAI | Arize Observe 2026OpenTelemetry GenAI Semantic Conventions: Native Support in Arize AXFind the traces that matter with Phoenix and PXIHow to Manage LLM Context Windows for AI AgentsWhy AI Agents Fail at Tasks They Already SolvedScaling Reinforcement Learning for AI Agents with Ray | Anyscale | Arize Observe 2026Humans vs. AI Agents: Who Should Handle What in Production?Prompt Learning: How We Made Claude Code 20% Better Without Changing the ModelArize Skills Demo: Instrument, Debug, and Evaluate Without Leaving Your Coding Agent
Arize AI |

How AI Ops Teams Will Manage Fleets of Agents | Arize

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER