Uploaded September 2026 | Updated November 2026, 17 hours ago
You fixed one AI agent failure. But did the change actually improve the entire agent, or did it break another workflow?
In this Arize AX session, learn how to test AI agents end to end using production traces, evaluation datasets, remote agents, and agent experiments.
Modern agents are more than prompts. They use tools, workflows, policies, external systems, and sometimes multiple sub-agents. A change that improves one behavior can easily introduce a regression somewhere else.
That is why agent testing needs to move beyond individual prompt experiments.
In this session, the Arize team shows how to:
• Turn production traces into realistic agent test cases
• Build regression datasets from common workflows and known failures
• Connect Arize AX to an agent running in your own environment
• Run the same dataset against a baseline and candidate agent configuration
• Test changes to prompts, models, tools, and action policies
• Evaluate complete agent behavior with multiple evals
• Compare safety and task completion side by side
• Drill into full traces when an experiment regresses
• Give product managers and evaluation teams a way to test agents without writing custom scripts
• Use coding agents to analyze large experiment datasets
• Connect production observability and offline experimentation into one feedback loop
The demo uses an e-commerce agent that can search for products, make purchases, and cancel orders.
The original agent has an unsafe behavior: when a user asks to cancel their most recent order, the agent immediately performs the action without requesting confirmation or the necessary details.
A new policy fixes that failure, but agent experiments reveal a side effect: the agent becomes too cautious in valid purchase workflows and overall task completion declines.
That is the core reason to test agents end to end. Fixing one trace does not necessarily mean you improved the system.
The session also explores the future of agent experimentation, including multi-turn session replay, simulated user interactions, sandbox testing, human approval, and agents that can identify failures, propose fixes, run experiments, and present validated changes for review.
The larger workflow becomes:
Production trace → failure → dataset → proposed fix → agent experiment → evals → comparison → ship or block.
That creates the foundation for continuously improving AI agents without relying on vibes or manually testing every change.
Chapters:
00:00 Why fixing one AI agent failure is not enough
02:40 Why prompt testing falls short for modern agents
06:10 From observability to end-to-end agent experimentation
10:15 Who needs agent experiments?
14:25 Why evals still matter as models improve
18:30 Prompt experiments vs. code experiments vs. agent experiments
23:35 Connecting production and experimentation
27:20 Demo: The unsafe e-commerce agent
31:25 Set up a remote agent in Arize AX
35:50 Build an AI agent regression dataset
39:35 Run baseline and candidate experiments
43:10 Compare safety and task completion evals
46:30 Finding a regression in another workflow
49:50 Why full agent traces matter
52:05 Building the agent quality feedback loop
55:15 The future: session replay and simulated users
58:20 Agents that diagnose, test, and validate their own fixes
1:01:10 How to get started with agent experiments
🔗 Explore Arize AX: arize.com/ax
🔗 Learn more about Arize: arize.com
🔔 Subscribe for more videos about AI agents, evaluation, observability, and AI engineering:
youtube.com/@arizeai?sub_confirmation=1
#AIAgents #AgentEvaluation #LLMEvals
You fixed one AI agent failure. But did the change actually improve the entire agent, or did it break another workflow?
In this Arize AX session, learn how to test AI agents end to end using production traces, evaluation datasets, remote agents, and agent experiments.
Modern agents are more than prompts. They use tools, workflows, policies, external systems, and sometimes multiple sub-agents. A change that improves one behavior can easily introduce a regression somewhere else.
That is why agent testing needs to move beyond individual prompt experiments.
In this session, the Arize team shows how to:
• Turn production traces into realistic agent test cases
• Build regression datasets from common workflows and known failures
• Connect Arize AX to an agent running in your own environment
• Run the same dataset against a baseline and candidate agent configuration
• Test changes to prompts, models, tools, and action policies
• Evaluate complete agent behavior with multiple evals
• Compare safety and task completion side by side
• Drill into full traces when an experiment regresses
• Give product managers and evaluation teams a way to test agents without writing custom scripts
• Use coding agents to analyze large experiment datasets
• Connect production observability and offline experimentation into one feedback loop
The demo uses an e-commerce agent that can search for products, make purchases, and cancel orders.
The original agent has an unsafe behavior: when a user asks to cancel their most recent order, the agent immediately performs the action without requesting confirmation or the necessary details.
A new policy fixes that failure, but agent experiments reveal a side effect: the agent becomes too cautious in valid purchase workflows and overall task completion declines.
That is the core reason to test agents end to end. Fixing one trace does not necessarily mean you improved the system.
The session also explores the future of agent experimentation, including multi-turn session replay, simulated user interactions, sandbox testing, human approval, and agents that can identify failures, propose fixes, run experiments, and present validated changes for review.
The larger workflow becomes:
Production trace → failure → dataset → proposed fix → agent experiment → evals → comparison → ship or block.
That creates the foundation for continuously improving AI agents without relying on vibes or manually testing every change.
Chapters:
00:00 Why fixing one AI agent failure is not enough
02:40 Why prompt testing falls short for modern agents
06:10 From observability to end-to-end agent experimentation
10:15 Who needs agent experiments?
14:25 Why evals still matter as models improve
18:30 Prompt experiments vs. code experiments vs. agent experiments
23:35 Connecting production and experimentation
27:20 Demo: The unsafe e-commerce agent
31:25 Set up a remote agent in Arize AX
35:50 Build an AI agent regression dataset
39:35 Run baseline and candidate experiments
43:10 Compare safety and task completion evals
46:30 Finding a regression in another workflow
49:50 Why full agent traces matter
52:05 Building the agent quality feedback loop
55:15 The future: session replay and simulated users
58:20 Agents that diagnose, test, and validate their own fixes
1:01:10 How to get started with agent experiments
🔗 Explore Arize AX: arize.com/ax
🔗 Learn more about Arize: arize.com
🔔 Subscribe for more videos about AI agents, evaluation, observability, and AI engineering:
youtube.com/@arizeai?sub_confirmation=1
#AIAgents #AgentEvaluation #LLMEvals










