Uploaded February 2026 | Updated November 2026, 1 day ago
Building AI systems that actually work in production is harder than it sounds. We built Alyx, Arize's agent for AX, and it broke in ways we didn't expect. This post is about what broke, what surprised us, and the patterns that actually worked -- with enough implementation detail that you can reuse them.
Alyx is an AI assistant that helps people use Arize AX, a platform for observing and evaluating AI systems. Users ask questions in natural language:
"What's the bottleneck in this trace?"
"Which experiment had better accuracy?"
"Why did this eval score drop?"
Alyx is an LLM-powered agent with tools that map to product features -- trace analysis, experiment comparison, dataset operations, eval scoring. Under the hood, it's an orchestrator loop: the LLM receives a user question, decides which tools to call, processes the results, and either calls more tools or responds.
In this video, Laurie Voss covers four lessons we learned -- mostly the hard way.
Check out Alyx: arize.com/docs/ax/alyx/arize-copilot
Building AI systems that actually work in production is harder than it sounds. We built Alyx, Arize's agent for AX, and it broke in ways we didn't expect. This post is about what broke, what surprised us, and the patterns that actually worked -- with enough implementation detail that you can reuse them.
Alyx is an AI assistant that helps people use Arize AX, a platform for observing and evaluating AI systems. Users ask questions in natural language:
"What's the bottleneck in this trace?"
"Which experiment had better accuracy?"
"Why did this eval score drop?"
Alyx is an LLM-powered agent with tools that map to product features -- trace analysis, experiment comparison, dataset operations, eval scoring. Under the hood, it's an orchestrator loop: the LLM receives a user question, decides which tools to call, processes the results, and either calls more tools or responds.
In this video, Laurie Voss covers four lessons we learned -- mostly the hard way.
Check out Alyx: arize.com/docs/ax/alyx/arize-copilot










