Uploaded August 2025 | Updated September 2026, 3 weeks ago
This lecture brings together Arize, Google Cloud, and Wayfair to explore the latest in AI observability, evaluation, and agent development.
Speakers showcase workflows for tracing, LLM-as-a-judge evaluations, and feedback loops, with live demos highlighting how to detect, debug, and improve AI systems in both development and production.
Wayfair shares real-world successes using these tools for catalog enrichment, supply chain optimization, and multi-agent frameworks, while Google Cloud outlines Gemini’s capabilities and responsible AI approach.
Timestamps
0:00 – Intro & Agenda – Jason (CEO, Arize) opens event, outlines focus on AI observability & evaluation, agenda with Google Cloud + Wayfair.
2:07 – What is Evaluation? – LLM-as-judge, moving beyond CSVs, 3 pillars: tracing, evals, experimentation.
4:48 – Arize Evolution – From observability to evals & prompt IDE, intro to “Alex” AI assistant.
6:21 – Platform Architecture – Dev ↔ Prod feedback loop, tracing, eval spectrum (LLM-judge, code checks, human annotations).
12:24 – Eval Workflows – Dev playground (CSV) vs. online production evals.
14:10 – Live Demo – Tracing, span-level evals, fixing hallucinations, prompt changes, Prompt Hub, AI Assist.
26:06 – Google Cloud (Cindy) – Responsible AI, Gemini’s long context, reasoning, multimodality, low hallucination.
39:09 – Customer Stories – Wayfair catalog tagging, Viral Nation brand safety.
42:25 – Keys to Agents – Must be autonomous & optimized; tracing, evals, feedback loops.
43:03 – Dylan: Feedback-Driven Dev – Agent challenges, eval points (router, tools, decision path, sessions), English feedback, Prompt Learning SDK.
1:01:33 – Wayfair Panel – LLMOps early, agent examples, multi-agent use, LangChain, Google A2A, MCP security, LLM-as-a-jury.
1:20:00 – Closing – Community & customer support importance, networking.
Relevant links
Arize Enterprise Platform – arize.com
Arize Phoenix (Open-Source) – github.com/Arize-ai/phoenix
Google Cloud Vertex AI – cloud.google.com/vertex-ai
Model Context Protocol (MCP) – modelcontextprotocol.io
This lecture brings together Arize, Google Cloud, and Wayfair to explore the latest in AI observability, evaluation, and agent development.
Speakers showcase workflows for tracing, LLM-as-a-judge evaluations, and feedback loops, with live demos highlighting how to detect, debug, and improve AI systems in both development and production.
Wayfair shares real-world successes using these tools for catalog enrichment, supply chain optimization, and multi-agent frameworks, while Google Cloud outlines Gemini’s capabilities and responsible AI approach.
Timestamps
0:00 – Intro & Agenda – Jason (CEO, Arize) opens event, outlines focus on AI observability & evaluation, agenda with Google Cloud + Wayfair.
2:07 – What is Evaluation? – LLM-as-judge, moving beyond CSVs, 3 pillars: tracing, evals, experimentation.
4:48 – Arize Evolution – From observability to evals & prompt IDE, intro to “Alex” AI assistant.
6:21 – Platform Architecture – Dev ↔ Prod feedback loop, tracing, eval spectrum (LLM-judge, code checks, human annotations).
12:24 – Eval Workflows – Dev playground (CSV) vs. online production evals.
14:10 – Live Demo – Tracing, span-level evals, fixing hallucinations, prompt changes, Prompt Hub, AI Assist.
26:06 – Google Cloud (Cindy) – Responsible AI, Gemini’s long context, reasoning, multimodality, low hallucination.
39:09 – Customer Stories – Wayfair catalog tagging, Viral Nation brand safety.
42:25 – Keys to Agents – Must be autonomous & optimized; tracing, evals, feedback loops.
43:03 – Dylan: Feedback-Driven Dev – Agent challenges, eval points (router, tools, decision path, sessions), English feedback, Prompt Learning SDK.
1:01:33 – Wayfair Panel – LLMOps early, agent examples, multi-agent use, LangChain, Google A2A, MCP security, LLM-as-a-jury.
1:20:00 – Closing – Community & customer support importance, networking.
Relevant links
Arize Enterprise Platform – arize.com
Arize Phoenix (Open-Source) – github.com/Arize-ai/phoenix
Google Cloud Vertex AI – cloud.google.com/vertex-ai
Model Context Protocol (MCP) – modelcontextprotocol.io
