Uploaded July 2025 | Updated September 2026, 3 weeks ago
Generative AI apps introduce a minefield of observability challenges—from unpredictable outputs to fragile integration points and shifting APIs. In this fast-paced walkthrough, Adrian Cole shares a portable, step-by-step approach to debugging and evaluating LLM systems, starting with tools you already know and ending with advanced techniques using Arize Phoenix.
Chapters
00:00 – Intro: Traditional vs GenAI Observability
01:45 – Debugging LLMs: Start With Your Laptop
03:00 – It’s Just a Web Service: Demystifying LLM APIs
05:20 – Observability Tools: Using mitmproxy & OpenTelemetry
07:30 – Integration Testing for LLMs: Handling Flaky Responses
10:05 – Recording & Replaying LLM Calls (VCR, Knock)
12:15 – Offline Evals With Phoenix + Unit Test Runners
14:00 – Writing Custom Metrics: Example With Hallucination
16:00 – Human-in-the-Loop Feedback + Eval Backfilling
18:00 – Combining Elastic & Phoenix: Traces Across Systems
20:00 – Tips on Guardrails, Redaction, and Secrets
22:10 – Q&A: Prompt Eval, Redaction, and Guardrail Systems
Learn more about Phoenix:
arize.com/docs/phoenix
Generative AI apps introduce a minefield of observability challenges—from unpredictable outputs to fragile integration points and shifting APIs. In this fast-paced walkthrough, Adrian Cole shares a portable, step-by-step approach to debugging and evaluating LLM systems, starting with tools you already know and ending with advanced techniques using Arize Phoenix.
Chapters
00:00 – Intro: Traditional vs GenAI Observability
01:45 – Debugging LLMs: Start With Your Laptop
03:00 – It’s Just a Web Service: Demystifying LLM APIs
05:20 – Observability Tools: Using mitmproxy & OpenTelemetry
07:30 – Integration Testing for LLMs: Handling Flaky Responses
10:05 – Recording & Replaying LLM Calls (VCR, Knock)
12:15 – Offline Evals With Phoenix + Unit Test Runners
14:00 – Writing Custom Metrics: Example With Hallucination
16:00 – Human-in-the-Loop Feedback + Eval Backfilling
18:00 – Combining Elastic & Phoenix: Traces Across Systems
20:00 – Tips on Guardrails, Redaction, and Secrets
22:10 – Q&A: Prompt Eval, Redaction, and Guardrail Systems
Learn more about Phoenix:
arize.com/docs/phoenix
