Uploaded November 2026 | Updated November 2026, 3 hours ago
Why should an AI coding agent read thousands of spans just to count errors?
In this technical deep dive, the Arize Phoenix team shows how Phoenix MCP uses SQL and Code Mode to analyze AI agent traces at approximately 17x lower average cost than traditional retrieval-only MCP tools in our benchmark.
Join Nancy, Roger, and Maura from Arize as they explore how the built-in Phoenix MCP server connects coding agents like Claude Code, Cursor, and Codex directly to observability data, making it easier to investigate production errors, analyze traces, and debug AI applications without wasting tokens on unnecessary tool calls.
Instead of fetching thousands of spans page by page, Phoenix MCP lets coding agents use read-only SQL to aggregate trace data. With Code Mode, agents can also run Python inside an isolated sandbox, process intermediate results, and return only the information the model needs.
What you'll learn:
• Why traditional MCP retrieval tools become expensive when analyzing thousands of traces
• How Phoenix MCP connects coding agents to traces, spans, and annotations
• How Code Mode reduces token usage and context window overhead
• How Phoenix runs agent-generated Python in an isolated sandbox
• How agents discover database schemas and generate SQL queries
• How Phoenix supports trace analysis with SQLite and PostgreSQL
• How Phoenix protects data with read-only SQL, query validation, and execution limits
• What the Phoenix MCP benchmarks revealed about cost, efficiency, and accuracy
• How to connect Claude Code to Phoenix MCP
• How to investigate failed tool calls and recurring errors using SQL
The benchmark: SQL + Code Mode vs. traditional MCP retrieval
We compared retrieval-only MCP tools against SQL and Code Mode using the same agent, model, questions, and trace data.
The benchmark included eight trace-analysis questions, a project with approximately 3,600 spans, and three runs per question.
Across the eight questions, the SQL-based approach achieved:
• Approximately 17x lower average cost per question
• 6.5 steps per question, compared with 64.5 using retrieval-only tools
• More reliable aggregation results, avoiding counting errors observed in some retrieval-based runs
For one question, "How many traces hit a high-impact error?", the difference was dramatic.
Traditional MCP retrieval:
• 89 turns
• 14 minutes
• $10.35
Phoenix MCP with SQL + Code Mode:
• 7 turns
• 33 seconds
• $0.23
Both approaches found the correct answer of 99 traces, but SQL allowed the database to do the counting instead of making the LLM process thousands of spans.
The takeaway? Use SQL to find patterns across many traces, then use retrieval to investigate individual failures. Let the database handle aggregation while your AI agent focuses on interpreting the results.
Chapters
00:00 Why AI Agents Shouldn't Read Thousands of Spans
01:38 Introducing Phoenix MCP
04:26 Why Traditional MCP Trace Retrieval Gets Expensive
06:38 Benchmark: 89 Turns, 14 Minutes, and $10.35
07:42 How the Phoenix MCP Server Works
09:56 What Is Code Mode?
11:34 Running Python in the Phoenix Sandbox
13:48 SQL Tools: Schema Discovery and Query Execution
15:34 How Agents Query Traces with SQL
17:54 Keeping Agent-Generated SQL Safe
20:12 How We Designed the Phoenix MCP Benchmark
22:16 Benchmark Results: 17x Lower Average Cost
24:00 How to Set Up Phoenix MCP with Claude Code
25:42 Demo: Debugging Failed Tool Calls with SQL
26:30 When to Use SQL vs. Trace Retrieval
28:50 Q&A: Security and Benchmark Methodology
31:24 Resources and Closing
Get started with Arize Phoenix
Phoenix is an open source AI observability and evaluation platform with a built-in MCP server.
To get started, you'll need Phoenix 20.2 or later, trace data in your Phoenix instance, and an MCP-compatible coding agent.
Open the MCP settings in Phoenix, copy the connection command for your coding agent, and start asking questions about your traces.
🔗 Explore Arize Phoenix: phoenix.arize.com
🔗 Phoenix on GitHub: github.com/Arize-ai/phoenix
🔗 Phoenix documentation: arize.com/docs/phoenix
🔗 Learn more about Arize: arize.com
🔗 Upcoming Arize events: luma.com/arizeai
🔔 Subscribe for more technical content on AI agents, MCP, LLM evaluation, observability, and open source AI: youtube.com/@arizeai?sub_confirmation=1
#ArizePhoenix #MCP #AIAgents
Why should an AI coding agent read thousands of spans just to count errors?
In this technical deep dive, the Arize Phoenix team shows how Phoenix MCP uses SQL and Code Mode to analyze AI agent traces at approximately 17x lower average cost than traditional retrieval-only MCP tools in our benchmark.
Join Nancy, Roger, and Maura from Arize as they explore how the built-in Phoenix MCP server connects coding agents like Claude Code, Cursor, and Codex directly to observability data, making it easier to investigate production errors, analyze traces, and debug AI applications without wasting tokens on unnecessary tool calls.
Instead of fetching thousands of spans page by page, Phoenix MCP lets coding agents use read-only SQL to aggregate trace data. With Code Mode, agents can also run Python inside an isolated sandbox, process intermediate results, and return only the information the model needs.
What you'll learn:
• Why traditional MCP retrieval tools become expensive when analyzing thousands of traces
• How Phoenix MCP connects coding agents to traces, spans, and annotations
• How Code Mode reduces token usage and context window overhead
• How Phoenix runs agent-generated Python in an isolated sandbox
• How agents discover database schemas and generate SQL queries
• How Phoenix supports trace analysis with SQLite and PostgreSQL
• How Phoenix protects data with read-only SQL, query validation, and execution limits
• What the Phoenix MCP benchmarks revealed about cost, efficiency, and accuracy
• How to connect Claude Code to Phoenix MCP
• How to investigate failed tool calls and recurring errors using SQL
The benchmark: SQL + Code Mode vs. traditional MCP retrieval
We compared retrieval-only MCP tools against SQL and Code Mode using the same agent, model, questions, and trace data.
The benchmark included eight trace-analysis questions, a project with approximately 3,600 spans, and three runs per question.
Across the eight questions, the SQL-based approach achieved:
• Approximately 17x lower average cost per question
• 6.5 steps per question, compared with 64.5 using retrieval-only tools
• More reliable aggregation results, avoiding counting errors observed in some retrieval-based runs
For one question, "How many traces hit a high-impact error?", the difference was dramatic.
Traditional MCP retrieval:
• 89 turns
• 14 minutes
• $10.35
Phoenix MCP with SQL + Code Mode:
• 7 turns
• 33 seconds
• $0.23
Both approaches found the correct answer of 99 traces, but SQL allowed the database to do the counting instead of making the LLM process thousands of spans.
The takeaway? Use SQL to find patterns across many traces, then use retrieval to investigate individual failures. Let the database handle aggregation while your AI agent focuses on interpreting the results.
Chapters
00:00 Why AI Agents Shouldn't Read Thousands of Spans
01:38 Introducing Phoenix MCP
04:26 Why Traditional MCP Trace Retrieval Gets Expensive
06:38 Benchmark: 89 Turns, 14 Minutes, and $10.35
07:42 How the Phoenix MCP Server Works
09:56 What Is Code Mode?
11:34 Running Python in the Phoenix Sandbox
13:48 SQL Tools: Schema Discovery and Query Execution
15:34 How Agents Query Traces with SQL
17:54 Keeping Agent-Generated SQL Safe
20:12 How We Designed the Phoenix MCP Benchmark
22:16 Benchmark Results: 17x Lower Average Cost
24:00 How to Set Up Phoenix MCP with Claude Code
25:42 Demo: Debugging Failed Tool Calls with SQL
26:30 When to Use SQL vs. Trace Retrieval
28:50 Q&A: Security and Benchmark Methodology
31:24 Resources and Closing
Get started with Arize Phoenix
Phoenix is an open source AI observability and evaluation platform with a built-in MCP server.
To get started, you'll need Phoenix 20.2 or later, trace data in your Phoenix instance, and an MCP-compatible coding agent.
Open the MCP settings in Phoenix, copy the connection command for your coding agent, and start asking questions about your traces.
🔗 Explore Arize Phoenix: phoenix.arize.com
🔗 Phoenix on GitHub: github.com/Arize-ai/phoenix
🔗 Phoenix documentation: arize.com/docs/phoenix
🔗 Learn more about Arize: arize.com
🔗 Upcoming Arize events: luma.com/arizeai
🔔 Subscribe for more technical content on AI agents, MCP, LLM evaluation, observability, and open source AI: youtube.com/@arizeai?sub_confirmation=1
#ArizePhoenix #MCP #AIAgents










