Uploaded May 2026 | Updated September 2026, 2 weeks ago
This is part three of our deep dive series on how we built Alyx, our AI engineering agent, and centers on testing and evals. AI agents don’t fail like traditional software. A small prompt edit, tool description change, or model upgrade can quietly break multi-step behavior.
• Related blog post [part three]: Why testing agents is non-negotiable
arize.com/blog/why-testing-ai-agents-is-non-negotiable
• Part one: How to build planning into your agent
arize.com/blog/how-to-build-planning-into-your-agent
• Part two: How to manage LLM context windows in agents
arize.com/blog/how-to-manage-llm-context-windows-for-ai-agents
In this Alyx deep dive, Sally Ann and Jack walk through how the Arize team tests production AI agents using real traces, golden datasets, experiments, and LLM-as-judge evals. You’ll see how a failing Alyx trace becomes a test case, how expectations are written in natural language, and how experiments help track regressions over time.
We walk through the eval framework we built after outgrowing a Google Doc, and why we now consider it non-negotiable before shipping anything to production.
This video covers:
• Why unit-test-style assertions break down for agents
• How to turn production traces into reusable test cases
• How LLM-as-judge evals validate behavior without brittle exact matching
• A test-driven development workflow for fixing agent regressions
• Why experiments are useful for tracking eval results over time
• How this workflow can plug into CI/CD before prompt, tool, or model changes ship
Learn more about Arize AX: arize.com
#AIagents #AIEvals #LLMEvals #AgentTesting #LLMObservability #ArizeAI #ArizeAX #ProductionTraces #TestDrivenDevelopment #TDDforAI #LLMasJudge #PromptEngineering #AIEngineering #AgentDebugging #CICDforAI #GoldenDatasets #EvalDatasets #AIObservability #MachineLearning #GenerativeAI
— CHAPTERS —
00:00 Why testing AI agents is hard
00:35 Early days: manual testing and why it breaks
01:15 Why small prompt/tool changes cause regressions
02:00 Debugging a failing Alyx trace
02:50 Turning a trace into a test case
03:40 Writing flexible eval expectations (LLM-as-judge)
04:45 Running the agent test harness
05:40 Logging results as an experiment in Arize
06:20 Why production traces improve team collaboration
07:10 Running the eval and analyzing failure
08:20 Finding the root cause in the system prompt
09:10 Fixing the bug and rerunning tests
10:10 Still failing: iterating with TDD
11:10 Building a regression test suite over time
12:00 CI/CD integration and final takeaways
This is part three of our deep dive series on how we built Alyx, our AI engineering agent, and centers on testing and evals. AI agents don’t fail like traditional software. A small prompt edit, tool description change, or model upgrade can quietly break multi-step behavior.
• Related blog post [part three]: Why testing agents is non-negotiable
arize.com/blog/why-testing-ai-agents-is-non-negotiable
• Part one: How to build planning into your agent
arize.com/blog/how-to-build-planning-into-your-agent
• Part two: How to manage LLM context windows in agents
arize.com/blog/how-to-manage-llm-context-windows-for-ai-agents
In this Alyx deep dive, Sally Ann and Jack walk through how the Arize team tests production AI agents using real traces, golden datasets, experiments, and LLM-as-judge evals. You’ll see how a failing Alyx trace becomes a test case, how expectations are written in natural language, and how experiments help track regressions over time.
We walk through the eval framework we built after outgrowing a Google Doc, and why we now consider it non-negotiable before shipping anything to production.
This video covers:
• Why unit-test-style assertions break down for agents
• How to turn production traces into reusable test cases
• How LLM-as-judge evals validate behavior without brittle exact matching
• A test-driven development workflow for fixing agent regressions
• Why experiments are useful for tracking eval results over time
• How this workflow can plug into CI/CD before prompt, tool, or model changes ship
Learn more about Arize AX: arize.com
#AIagents #AIEvals #LLMEvals #AgentTesting #LLMObservability #ArizeAI #ArizeAX #ProductionTraces #TestDrivenDevelopment #TDDforAI #LLMasJudge #PromptEngineering #AIEngineering #AgentDebugging #CICDforAI #GoldenDatasets #EvalDatasets #AIObservability #MachineLearning #GenerativeAI
— CHAPTERS —
00:00 Why testing AI agents is hard
00:35 Early days: manual testing and why it breaks
01:15 Why small prompt/tool changes cause regressions
02:00 Debugging a failing Alyx trace
02:50 Turning a trace into a test case
03:40 Writing flexible eval expectations (LLM-as-judge)
04:45 Running the agent test harness
05:40 Logging results as an experiment in Arize
06:20 Why production traces improve team collaboration
07:10 Running the eval and analyzing failure
08:20 Finding the root cause in the system prompt
09:10 Fixing the bug and rerunning tests
10:10 Still failing: iterating with TDD
11:10 Building a regression test suite over time
12:00 CI/CD integration and final takeaways
