Uploaded January 2026 | Updated September 2026, 2 weeks ago
In this episode, we’re diving into observability for Generative AI apps. AI helps us write code and monitor applications in production - but how do we observe the AI itself? And how do we make sense of complex, non-deterministic AI systems?
🎙️ We’re joined by two great guests: Ishan Jain, working on GenAI observability and Luccas Quadros, working on Grafana Assistant.
Together, they bring both platform-level insights and real-world perspectives.
🚀 We’ll explore the fast-evolving space of observability for generative AI apps, including:
- How non-deterministic responses change the way we think about observability
- Recording prompts and completions in telemetry: when it’s useful, when it’s not, and how to do it safely
- Using automated evaluations to understand whether things actually work
- Monitoring agentic workflows and multi-step reasoning
- What OTel GenAI group works on
💬 Join us live for an open conversation! Bring your questions, opinions, and hot takes - it’s going to be a fun one 🤖
TIMESTAMPS
00:00:00 Introductions and Overview of the Call
00:03:15 Guests Introductions: Ishan and Lucas
00:05:00 Announcement: Final Chance to Register for Otel Unplugged
00:06:30 Discussion: What is Generative AI?
00:10:45 The Evolution of AI and Observability Techniques
00:15:30 Challenges in Observing Generative AI Applications
00:20:00 Importance of Latency in AI Interactions
00:25:00 Demo: Observability for AI Agent Execution
00:32:00 Discussing Costs Associated with AI Model Evaluations
00:40:00 Evaluating AI Responses with LLM as Judge
00:50:00 Closing Thoughts and Resources for Getting Started with Observability
In this episode, we’re diving into observability for Generative AI apps. AI helps us write code and monitor applications in production - but how do we observe the AI itself? And how do we make sense of complex, non-deterministic AI systems?
🎙️ We’re joined by two great guests: Ishan Jain, working on GenAI observability and Luccas Quadros, working on Grafana Assistant.
Together, they bring both platform-level insights and real-world perspectives.
🚀 We’ll explore the fast-evolving space of observability for generative AI apps, including:
- How non-deterministic responses change the way we think about observability
- Recording prompts and completions in telemetry: when it’s useful, when it’s not, and how to do it safely
- Using automated evaluations to understand whether things actually work
- Monitoring agentic workflows and multi-step reasoning
- What OTel GenAI group works on
💬 Join us live for an open conversation! Bring your questions, opinions, and hot takes - it’s going to be a fun one 🤖
TIMESTAMPS
00:00:00 Introductions and Overview of the Call
00:03:15 Guests Introductions: Ishan and Lucas
00:05:00 Announcement: Final Chance to Register for Otel Unplugged
00:06:30 Discussion: What is Generative AI?
00:10:45 The Evolution of AI and Observability Techniques
00:15:30 Challenges in Observing Generative AI Applications
00:20:00 Importance of Latency in AI Interactions
00:25:00 Demo: Observability for AI Agent Execution
00:32:00 Discussing Costs Associated with AI Model Evaluations
00:40:00 Evaluating AI Responses with LLM as Judge
00:50:00 Closing Thoughts and Resources for Getting Started with Observability
