Uploaded February 2026 | Updated November 2026, 3 hours ago
Arize AX Enablement session on building evaluators! In this hands-on session, we show you how to create LLM-as-a-Judge and code-based evaluators, scope them across spans, traces, and sessions, and apply filters to evaluate on exactly the data that matters. Designed for both technical and non-technical attendees - this session focuses on real examples that make evaluators approachable and actionable for everyone.
Learn how OpenTelemetry tracing works for AI apps—spans, traces, sessions, exporters, span processors, and tracer providers—and how OpenInference semantic conventions + auto-instrumentation make LLM/agent workflows observable in Arize AX. We close with advanced topics like context propagation, sampling, and OTEL collectors.
0:00 Intro
0:49 Why observability (trust + silent failures)
4:07 OpenTelemetry overview
7:17 OpenInference overview
8:21 Signals (traces, metrics, & logs)
10:15 Span structure
16:31 OTel components (resource/exporter/processor/provider)
18:17 Resource
21:48 Exporters
31:32 Span processors
35:05 Tracer provider
38:53 Register function (arize.otel)
39:24 ArizeRoutingProcessor
41:25 OpenInference Semantic Conventions
44:59 How to Instrument
45:51 Autoinstrumentation
46:58 Manual instrumentation
51:17 OpenInference Context Managers
52:15 Hybrid instrumentation
53:54 Custom Span Processors
57:26 Context propagation
1:05:36 Sampling
1:12:43 Collector
Speakers:
Sofia Jakovcevic: linkedin.com/in/sofia-jakovcevic
Lucas Moehlenbrock: linkedin.com/in/lucas-moehlenbrock-b42174130
Arize docs:
Tracing overview: arize.com/docs/ax/observe/tracing
Quickstart tracing: arize.com/docs/ax/quickstarts/quickstart-tracing
OpenTelemetry integration (arize-otel): arize.com/docs/ax/integrations/opentelemetry/opentelemetry-arize-otel
OpenInference helpers: arize.com/docs/ax/observe/tracing/configure-tracing-options/instrument-with-openinference-helpers
Advanced OTEL examples: arize.com/docs/ax/observe/tracing/configure/advanced-tracing-otel-examples
Arize community events:
arize.com/community
Arize AX Enablement session on building evaluators! In this hands-on session, we show you how to create LLM-as-a-Judge and code-based evaluators, scope them across spans, traces, and sessions, and apply filters to evaluate on exactly the data that matters. Designed for both technical and non-technical attendees - this session focuses on real examples that make evaluators approachable and actionable for everyone.
Learn how OpenTelemetry tracing works for AI apps—spans, traces, sessions, exporters, span processors, and tracer providers—and how OpenInference semantic conventions + auto-instrumentation make LLM/agent workflows observable in Arize AX. We close with advanced topics like context propagation, sampling, and OTEL collectors.
0:00 Intro
0:49 Why observability (trust + silent failures)
4:07 OpenTelemetry overview
7:17 OpenInference overview
8:21 Signals (traces, metrics, & logs)
10:15 Span structure
16:31 OTel components (resource/exporter/processor/provider)
18:17 Resource
21:48 Exporters
31:32 Span processors
35:05 Tracer provider
38:53 Register function (arize.otel)
39:24 ArizeRoutingProcessor
41:25 OpenInference Semantic Conventions
44:59 How to Instrument
45:51 Autoinstrumentation
46:58 Manual instrumentation
51:17 OpenInference Context Managers
52:15 Hybrid instrumentation
53:54 Custom Span Processors
57:26 Context propagation
1:05:36 Sampling
1:12:43 Collector
Speakers:
Sofia Jakovcevic: linkedin.com/in/sofia-jakovcevic
Lucas Moehlenbrock: linkedin.com/in/lucas-moehlenbrock-b42174130
Arize docs:
Tracing overview: arize.com/docs/ax/observe/tracing
Quickstart tracing: arize.com/docs/ax/quickstarts/quickstart-tracing
OpenTelemetry integration (arize-otel): arize.com/docs/ax/integrations/opentelemetry/opentelemetry-arize-otel
OpenInference helpers: arize.com/docs/ax/observe/tracing/configure-tracing-options/instrument-with-openinference-helpers
Advanced OTEL examples: arize.com/docs/ax/observe/tracing/configure/advanced-tracing-otel-examples
Arize community events:
arize.com/community










![How to Build the Right Evals for AI Agents | Arize Phoenix
AI agent evaluation can feel overwhelming. Agents use tools, skills, memory, files, and multi-turn conversations, creating far more failure points than a traditional LLM application.
In this Arize:Observe 2026 session, Elizabeth Hutton, Senior Software Engineer for Evals at Arize, shares what the Phoenix team learned while building and evaluating Pixie, an AI engineering agent inside the open source Arize Phoenix platform.
Rather than trying to design a complete evaluation suite upfront, the team built evals gradually as the agent gained new capabilities. Elizabeth explains how tracing, targeted capability evals, regression testing, production feedback, and trace analysis work together to create a practical agent improvement loop.
The session covers:
• Why tracing should come before formal evaluations
• How traces become the source of truth for debugging agents
• How to create small capability evals for tools, skills, and output formatting
• Using synthetic data to cover edge cases, typos, and negative examples
• Storing eval datasets and harnesses alongside application code
• Running agent evals and experiments with coding agents
• Turning targeted evals into regression checks in CI
• Combining narrow evals with end-to-end Playwright tests
• Why synthetic first-turn queries fail to represent real agent behavior
• Building realistic multi-turn datasets from production traces
• Using user feedback, LLM-as-a-judge, code evaluators, and trajectory evals
• Reviewing traces to discover failure modes you did not anticipate
• Moving production failures into development and regression datasets
• Closing the loop between tracing, experiments, implementation, and evals
The central lesson: evaluation is a development practice, not a one-time project. Start with tracing, add narrow evals as capabilities emerge, and expand the suite using failures discovered in production. :contentReference[oaicite:0]{index=0}
Chapters:
00:00 Why AI agent evals feel overwhelming
01:09 Meet Pixie, Phoenix’s AI engineering agent
02:33 Agent architecture and the growing failure surface
03:34 Start with tracing
04:53 Add narrow capability evals
06:52 Turn capability evals into regression tests
08:53 Why synthetic evaluation data falls short
11:03 Discovery evals for production agents
12:04 User feedback and automated evaluations
12:58 Trace review and agent error analysis
14:33 Closing the agent improvement loop
14:59 Lessons learned from building Pixie
16:11 Open source evals in Arize Phoenix
🔗 Explore Arize Phoenix: https://phoenix.arize.com
🔗 View Phoenix on GitHub: https://github.com/Arize-ai/phoenix
🔗 Read the documentation: https://arize.com/docs/phoenix
🔔 Subscribe for more videos about AI agents, LLM evaluation, observability, and open source AI:
https://www.youtube.com/@arizeai?sub_confirmation=1
#ArizePhoenix #LLMEvals #AIAgents How to Build the Right Evals for AI Agents | Arize Phoenix](https://i.ytimg.com/vi/jCE3tDDcuKE/mqdefault.jpg)