Uploaded October 2026 | Updated November 2026, 5 days ago
How do you know whether a change to your AI agent actually makes it better? And what if your agent completes the task correctly but wastes time, tokens, and tool calls along the way?
In this Arize webinar, Nancy Chauhan and Elizabeth Hutton from the Phoenix team demonstrate how to benchmark AI agents with Harbor and Arize Phoenix, compare agent configurations, and catch inefficiencies before they reach production.
Harbor is an open source framework for running agent benchmarks in reproducible, sandboxed environments. Arize Phoenix captures the resulting experiments, evaluations, scores, and agent traces so teams can compare performance and investigate failures.
The live demo shows how the Phoenix team benchmarked five agent configurations across 18 tasks, comparing Claude Code, Codex, and Phoenix's built-in agent, PXI.
The team discovered that Codex was repeatedly loading an unnecessary skill. By changing the skill description, they reduced the benchmarked agent's cost by 35% without degrading correctness.
In this webinar, you'll learn how to:
• Build repeatable AI agent benchmarks with Harbor
• Run agents in isolated Docker or Daytona sandboxes
• Define benchmark tasks, environments, and verifiers
• Evaluate agent outputs and the final state of an environment
• Compare Claude Code, Codex, and PXI on the same tasks
• Test different agent interfaces, including MCP servers, CLIs, and skills
• Capture Harbor benchmark runs as Phoenix datasets and experiments
• Version datasets and compare agent configurations over time
• Inspect agent trajectories, tool calls, token usage, and cost
• Distinguish agent failures from infrastructure failures
• Identify unnecessary tool calls and inefficient agent behavior
• Run repeated trials to measure reliability
• Use benchmark results to detect regressions and improve agent performance
The key takeaway: a successful agent run is not necessarily an efficient one. By combining reproducible benchmarks with detailed tracing, engineering teams can understand not just whether an agent completed a task, but how it got there and what could be improved.
Chapters:
00:00 Did your AI agent actually get better?
00:40 Benchmarking AI agents with Harbor + Phoenix
04:10 Why offline agent benchmarks matter
05:50 Testing agents in realistic environments
07:56 How Harbor runs sandboxed benchmarks
09:44 Building benchmark tasks and environments
12:18 Verifying agent outcomes and measuring efficiency
13:28 Designing implementation-agnostic benchmarks
14:44 How Phoenix captures Harbor benchmark results
17:06 Live demo: Benchmarking agents with Phoenix
18:06 Comparing Claude Code, Codex, and Pixie
20:26 Analyzing agent traces and tool calls
21:54 Finding unnecessary skill invocations
23:12 Reducing agent costs by 35% without sacrificing quality
25:00 Continuous benchmarking and regression testing
27:16 Getting started with Harbor + Arize Phoenix
28:06 Q&A: Offline vs. online agent evaluations
28:48 Q&A: Benchmarking LangGraph and production agents
30:12 Q&A: Measuring real improvements vs. noise
31:04 Resources and closing
🔗 Read the technical guide: arize.com/blog/ai-agent-benchmarking-harbor-arize-phoenix
🔗 Explore Arize Phoenix: phoenix.arize.com
🔗 Phoenix on GitHub: github.com/Arize-ai/phoenix
🔗 Harbor on GitHub: github.com/harbor-framework/harbor
🔔 Subscribe for more technical sessions on AI agent evaluation, benchmarking, observability, and production AI engineering:
youtube.com/@arizeai?sub_confirmation=1
#AIAgents #AgentEvaluation #ArizePhoenix
How do you know whether a change to your AI agent actually makes it better? And what if your agent completes the task correctly but wastes time, tokens, and tool calls along the way?
In this Arize webinar, Nancy Chauhan and Elizabeth Hutton from the Phoenix team demonstrate how to benchmark AI agents with Harbor and Arize Phoenix, compare agent configurations, and catch inefficiencies before they reach production.
Harbor is an open source framework for running agent benchmarks in reproducible, sandboxed environments. Arize Phoenix captures the resulting experiments, evaluations, scores, and agent traces so teams can compare performance and investigate failures.
The live demo shows how the Phoenix team benchmarked five agent configurations across 18 tasks, comparing Claude Code, Codex, and Phoenix's built-in agent, PXI.
The team discovered that Codex was repeatedly loading an unnecessary skill. By changing the skill description, they reduced the benchmarked agent's cost by 35% without degrading correctness.
In this webinar, you'll learn how to:
• Build repeatable AI agent benchmarks with Harbor
• Run agents in isolated Docker or Daytona sandboxes
• Define benchmark tasks, environments, and verifiers
• Evaluate agent outputs and the final state of an environment
• Compare Claude Code, Codex, and PXI on the same tasks
• Test different agent interfaces, including MCP servers, CLIs, and skills
• Capture Harbor benchmark runs as Phoenix datasets and experiments
• Version datasets and compare agent configurations over time
• Inspect agent trajectories, tool calls, token usage, and cost
• Distinguish agent failures from infrastructure failures
• Identify unnecessary tool calls and inefficient agent behavior
• Run repeated trials to measure reliability
• Use benchmark results to detect regressions and improve agent performance
The key takeaway: a successful agent run is not necessarily an efficient one. By combining reproducible benchmarks with detailed tracing, engineering teams can understand not just whether an agent completed a task, but how it got there and what could be improved.
Chapters:
00:00 Did your AI agent actually get better?
00:40 Benchmarking AI agents with Harbor + Phoenix
04:10 Why offline agent benchmarks matter
05:50 Testing agents in realistic environments
07:56 How Harbor runs sandboxed benchmarks
09:44 Building benchmark tasks and environments
12:18 Verifying agent outcomes and measuring efficiency
13:28 Designing implementation-agnostic benchmarks
14:44 How Phoenix captures Harbor benchmark results
17:06 Live demo: Benchmarking agents with Phoenix
18:06 Comparing Claude Code, Codex, and Pixie
20:26 Analyzing agent traces and tool calls
21:54 Finding unnecessary skill invocations
23:12 Reducing agent costs by 35% without sacrificing quality
25:00 Continuous benchmarking and regression testing
27:16 Getting started with Harbor + Arize Phoenix
28:06 Q&A: Offline vs. online agent evaluations
28:48 Q&A: Benchmarking LangGraph and production agents
30:12 Q&A: Measuring real improvements vs. noise
31:04 Resources and closing
🔗 Read the technical guide: arize.com/blog/ai-agent-benchmarking-harbor-arize-phoenix
🔗 Explore Arize Phoenix: phoenix.arize.com
🔗 Phoenix on GitHub: github.com/Arize-ai/phoenix
🔗 Harbor on GitHub: github.com/harbor-framework/harbor
🔔 Subscribe for more technical sessions on AI agent evaluation, benchmarking, observability, and production AI engineering:
youtube.com/@arizeai?sub_confirmation=1
#AIAgents #AgentEvaluation #ArizePhoenix










