Uploaded July 2025 | Updated September 2026, 2 weeks ago
📌 Description
What really is an AI agent? Are multi-agent systems worth it — or overhyped? Will one framework rule them all, or will developers stay close to the metal?
🔥 In this super candid, spicy panel, top founders from Crew AI, LlamaIndex, Mastra and Leta dive into the evolving world of agent frameworks. They debate:
What defines an agent vs basic LLM plumbing?
-Are most “agent frameworks” just glorified abstractions?
-Where do protocols like MCP and ADA actually fit (or fail)?
-Is multi-agent orchestration practical, or a demo trap?
-Why agent evaluation (evals) might be overrated — and when it matters.
-What’s next: perpetual agents, no-code builders, dependable systems.
-Perfect for developers, architects, or anyone deep into the future of agentic AI.
⏱ Timestamps
00:02 - Intro: Top agent framework founders on stage (Crew AI, LlamaIndex, Mastra, Leta)
0:26 - The age-old question: What IS an agent? Each founder defines it
1:46 - From LLMs with agency, to closed-loop systems, to control & memory
2:48 - How the term "agent" evolved from 2015 robotics RL to modern LLMs
5:16 - What’s an agent framework? Types in the ecosystem: workflows, orchestration, SDK vs service
6:48 - Why many frameworks are just middleware — or libraries vs servers
8:06 - Spicy take: “LangChainJS sucked — that’s why we built Mastra.”
9:50 - On abstractions: conventions, feature overload, and moving fast
12:50 - Will there be one framework to rule them all, or language-specific stacks?
14:55 - Library vs service trade-offs; easy to swap libraries, harder with hosted state
17:48 - Companies move from playful prototyping to secure production — frameworks commoditize
19:40 - Multi-agent debate: prompt engineering, decomposition, microservices parallels
23:00 - When multi-agent is roleplay vs serious system design
24:35 - Most production use cases today still resemble controlled pipelines, not unconstrained agent swarms
25:40 - MCP vs ADA: why MCP solves tool calling, but ADA is a solution looking for a problem
28:00 - The politics of protocols & vendor lock-in, plus open standards
30:50 - On evals: are they a moat or hype? When customers actually start caring
34:35 - Parallel to research: benchmarking vs method building — vibes first, evals later
36:45 - The next year: stable, perpetual agents; non-techs building agents
#AIagents, #agentframeworks, #CrewAI, #LlamaIndex, #Mastra, #Letta, #MCP
📌 Description
What really is an AI agent? Are multi-agent systems worth it — or overhyped? Will one framework rule them all, or will developers stay close to the metal?
🔥 In this super candid, spicy panel, top founders from Crew AI, LlamaIndex, Mastra and Leta dive into the evolving world of agent frameworks. They debate:
What defines an agent vs basic LLM plumbing?
-Are most “agent frameworks” just glorified abstractions?
-Where do protocols like MCP and ADA actually fit (or fail)?
-Is multi-agent orchestration practical, or a demo trap?
-Why agent evaluation (evals) might be overrated — and when it matters.
-What’s next: perpetual agents, no-code builders, dependable systems.
-Perfect for developers, architects, or anyone deep into the future of agentic AI.
⏱ Timestamps
00:02 - Intro: Top agent framework founders on stage (Crew AI, LlamaIndex, Mastra, Leta)
0:26 - The age-old question: What IS an agent? Each founder defines it
1:46 - From LLMs with agency, to closed-loop systems, to control & memory
2:48 - How the term "agent" evolved from 2015 robotics RL to modern LLMs
5:16 - What’s an agent framework? Types in the ecosystem: workflows, orchestration, SDK vs service
6:48 - Why many frameworks are just middleware — or libraries vs servers
8:06 - Spicy take: “LangChainJS sucked — that’s why we built Mastra.”
9:50 - On abstractions: conventions, feature overload, and moving fast
12:50 - Will there be one framework to rule them all, or language-specific stacks?
14:55 - Library vs service trade-offs; easy to swap libraries, harder with hosted state
17:48 - Companies move from playful prototyping to secure production — frameworks commoditize
19:40 - Multi-agent debate: prompt engineering, decomposition, microservices parallels
23:00 - When multi-agent is roleplay vs serious system design
24:35 - Most production use cases today still resemble controlled pipelines, not unconstrained agent swarms
25:40 - MCP vs ADA: why MCP solves tool calling, but ADA is a solution looking for a problem
28:00 - The politics of protocols & vendor lock-in, plus open standards
30:50 - On evals: are they a moat or hype? When customers actually start caring
34:35 - Parallel to research: benchmarking vs method building — vibes first, evals later
36:45 - The next year: stable, perpetual agents; non-techs building agents
#AIagents, #agentframeworks, #CrewAI, #LlamaIndex, #Mastra, #Letta, #MCP








![How to Build the Right Evals for AI Agents | Arize Phoenix
AI agent evaluation can feel overwhelming. Agents use tools, skills, memory, files, and multi-turn conversations, creating far more failure points than a traditional LLM application.
In this Arize:Observe 2026 session, Elizabeth Hutton, Senior Software Engineer for Evals at Arize, shares what the Phoenix team learned while building and evaluating Pixie, an AI engineering agent inside the open source Arize Phoenix platform.
Rather than trying to design a complete evaluation suite upfront, the team built evals gradually as the agent gained new capabilities. Elizabeth explains how tracing, targeted capability evals, regression testing, production feedback, and trace analysis work together to create a practical agent improvement loop.
The session covers:
• Why tracing should come before formal evaluations
• How traces become the source of truth for debugging agents
• How to create small capability evals for tools, skills, and output formatting
• Using synthetic data to cover edge cases, typos, and negative examples
• Storing eval datasets and harnesses alongside application code
• Running agent evals and experiments with coding agents
• Turning targeted evals into regression checks in CI
• Combining narrow evals with end-to-end Playwright tests
• Why synthetic first-turn queries fail to represent real agent behavior
• Building realistic multi-turn datasets from production traces
• Using user feedback, LLM-as-a-judge, code evaluators, and trajectory evals
• Reviewing traces to discover failure modes you did not anticipate
• Moving production failures into development and regression datasets
• Closing the loop between tracing, experiments, implementation, and evals
The central lesson: evaluation is a development practice, not a one-time project. Start with tracing, add narrow evals as capabilities emerge, and expand the suite using failures discovered in production. :contentReference[oaicite:0]{index=0}
Chapters:
00:00 Why AI agent evals feel overwhelming
01:09 Meet Pixie, Phoenix’s AI engineering agent
02:33 Agent architecture and the growing failure surface
03:34 Start with tracing
04:53 Add narrow capability evals
06:52 Turn capability evals into regression tests
08:53 Why synthetic evaluation data falls short
11:03 Discovery evals for production agents
12:04 User feedback and automated evaluations
12:58 Trace review and agent error analysis
14:33 Closing the agent improvement loop
14:59 Lessons learned from building Pixie
16:11 Open source evals in Arize Phoenix
đź”— Explore Arize Phoenix: https://phoenix.arize.com
đź”— View Phoenix on GitHub: https://github.com/Arize-ai/phoenix
đź”— Read the documentation: https://arize.com/docs/phoenix
đź”” Subscribe for more videos about AI agents, LLM evaluation, observability, and open source AI:
https://www.youtube.com/@arizeai?sub_confirmation=1
#ArizePhoenix #LLMEvals #AIAgents How to Build the Right Evals for AI Agents | Arize Phoenix](https://i.ytimg.com/vi/jCE3tDDcuKE/mqdefault.jpg)

![How to test AI agents with traces, evals, and CI/CD
This is part three of our deep dive series on how we built Alyx, our AI engineering agent, and centers on testing and evals. AI agents don’t fail like traditional software. A small prompt edit, tool description change, or model upgrade can quietly break multi-step behavior.
• Related blog post [part three]: Why testing agents is non-negotiable
https://arize.com/blog/why-testing-ai-agents-is-non-negotiable
• Part one: How to build planning into your agent
https://arize.com/blog/how-to-build-planning-into-your-agent
• Part two: How to manage LLM context windows in agents
https://arize.com/blog/how-to-manage-llm-context-windows-for-ai-agents/
In this Alyx deep dive, Sally Ann and Jack walk through how the Arize team tests production AI agents using real traces, golden datasets, experiments, and LLM-as-judge evals. You’ll see how a failing Alyx trace becomes a test case, how expectations are written in natural language, and how experiments help track regressions over time.
We walk through the eval framework we built after outgrowing a Google Doc, and why we now consider it non-negotiable before shipping anything to production.
This video covers:
• Why unit-test-style assertions break down for agents
• How to turn production traces into reusable test cases
• How LLM-as-judge evals validate behavior without brittle exact matching
• A test-driven development workflow for fixing agent regressions
• Why experiments are useful for tracking eval results over time
• How this workflow can plug into CI/CD before prompt, tool, or model changes ship
Learn more about Arize AX: https://arize.com/
#AIagents #AIEvals #LLMEvals #AgentTesting #LLMObservability #ArizeAI #ArizeAX #ProductionTraces #TestDrivenDevelopment #TDDforAI #LLMasJudge #PromptEngineering #AIEngineering #AgentDebugging #CICDforAI #GoldenDatasets #EvalDatasets #AIObservability #MachineLearning #GenerativeAI
— CHAPTERS —
00:00 Why testing AI agents is hard
00:35 Early days: manual testing and why it breaks
01:15 Why small prompt/tool changes cause regressions
02:00 Debugging a failing Alyx trace
02:50 Turning a trace into a test case
03:40 Writing flexible eval expectations (LLM-as-judge)
04:45 Running the agent test harness
05:40 Logging results as an experiment in Arize
06:20 Why production traces improve team collaboration
07:10 Running the eval and analyzing failure
08:20 Finding the root cause in the system prompt
09:10 Fixing the bug and rerunning tests
10:10 Still failing: iterating with TDD
11:10 Building a regression test suite over time
12:00 CI/CD integration and final takeaways How to test AI agents with traces, evals, and CI/CD](https://i.ytimg.com/vi/jjfnCAjdTkI/mqdefault.jpg)