How Lyft Builds Evals That Actually Matter in Production | Interrupt 26 @LangChain
How Lyft Builds Evals That Actually Matter in Production | Interrupt 26  @LangChain
Uploaded June 2026 | Updated November 2026, 1 day ago
Nick Ung, ML lead for safety and customer care at Lyft, breaks down how his team built an eval system that actually keeps pace with production AI agents at Interrupt, the agent conference by LangChain.

From offline simulation with mocked MCP outputs to LLM-as-judge rubrics tied to real task success criteria, Nick shares the mistakes they made early and the framework they built to close the loop between evaluation, annotation, and continuous improvement across 270K AI interactions per month.

How Lyft Builds Evals That Actually Matter in Production | Interrupt 26
0:00 Introduction and Lyft AI Assist overview
1:47 AI Assist journey and agent examples
3:35 Why eval matters at scale
4:18 Offline eval as a quality gate
6:03 Building a lightweight simulator
8:17 LLM as judge and the problem with scalar metrics
9:52 Task-based rubrics and actionable failure modes
11:28 Trusting your LLM judge
12:17 The rude awakening — when offline evals lie
14:14 How LangSmith manages the eval workflow
15:46 What's next for AI Assist

Extra resources:
• Everything we shipped at Interrupt: langchain.com/blog/interrupt-2026-overview
• Meet LangSmith Engine: langchain.com/blog/introducing-langsmith-engine
• About LangChain: langchain.com
How Lyft Builds Evals That Actually Matter in Production | Interrupt 26Why Do AI Agents Hallucinate?What is context engineering? Explained in 5 minutesInside Clays Eval Stack: 300M Agent Runs, One LangSmith PipelineNot Every Bug Engine Finds Is Worth FixingKeep model costs in check with LangSmith LLM GatewayBuilding a Harness with JevThe Agent for Your Agent.How Benchlings Team Actually Looks at AI Traces | Max AgencyBuilding Jarvis: JPMC’s Agent Building Assembly LineInside Managed Deep Agents: mda CLI, Context Hub, and Harbor Evals | Interrupt LondonWhat are Deep Agents? Explained fast
LangChain |

How Lyft Builds Evals That Actually Matter in Production | Interrupt 26

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER