Uploaded May 2025 | Updated September 2026, 4 hours ago
How do you ensure your AI systems actually do what you expect them to do? Leonard Tang takes us deep into the revolutionary world of AI evaluation with concrete techniques you can apply today. Learn how Haize Labs is transforming AI testing through "scaling judge-time compute" - stacking weaker models to effectively evaluate stronger ones. Leonard unpacks the game-changing Verdict library that outperforms frontier models by 10-20% while dramatically reducing costs. Discover practical insights on creating contrastive evaluation sets that extract maximum signal from human feedback, implementing debate-based judging systems, and building custom reward models that align with enterprise needs. The conversation reveals powerful nuggets like using randomized agent debates to achieve consensus and lightweight guardrail models that run alongside inference. Whether you're developing AI applications or simply fascinated by how we'll ensure increasingly powerful AI systems perform as expected, this episode delivers immediate value with techniques you can implement right away, philosophical perspectives on AI safety, and a glimpse into the future of evaluation that will fundamentally shape how AI evolves.
Learn more about Haize Labs! - haizelabs.com
Check out Verdict on GitHub - github.com/haizelabs/verdict
Chapters
0:00 Weaviate Podcast #121!
0:46 Welcome Leonard!
1:16 Founding Haize Labs
8:31 UX for Evals
17:01 Scaling Judge-Time Compute with Verdict
23:26 Debate Judges
26:06 Compute Scaling
28:50 Declarative Judge Pipelines
31:13 Custom Reward Models
37:20 Reasoning in Reward Models
39:20 Mechanistic Interpretability
45:30 Guardrails in Inference Pipelines
47:35 Can we control Superintelligence?
52:08 Exciting Directions for AI
How do you ensure your AI systems actually do what you expect them to do? Leonard Tang takes us deep into the revolutionary world of AI evaluation with concrete techniques you can apply today. Learn how Haize Labs is transforming AI testing through "scaling judge-time compute" - stacking weaker models to effectively evaluate stronger ones. Leonard unpacks the game-changing Verdict library that outperforms frontier models by 10-20% while dramatically reducing costs. Discover practical insights on creating contrastive evaluation sets that extract maximum signal from human feedback, implementing debate-based judging systems, and building custom reward models that align with enterprise needs. The conversation reveals powerful nuggets like using randomized agent debates to achieve consensus and lightweight guardrail models that run alongside inference. Whether you're developing AI applications or simply fascinated by how we'll ensure increasingly powerful AI systems perform as expected, this episode delivers immediate value with techniques you can implement right away, philosophical perspectives on AI safety, and a glimpse into the future of evaluation that will fundamentally shape how AI evolves.
Learn more about Haize Labs! - haizelabs.com
Check out Verdict on GitHub - github.com/haizelabs/verdict
Chapters
0:00 Weaviate Podcast #121!
0:46 Welcome Leonard!
1:16 Founding Haize Labs
8:31 UX for Evals
17:01 Scaling Judge-Time Compute with Verdict
23:26 Debate Judges
26:06 Compute Scaling
28:50 Declarative Judge Pipelines
31:13 Custom Reward Models
37:20 Reasoning in Reward Models
39:20 Mechanistic Interpretability
45:30 Guardrails in Inference Pipelines
47:35 Can we control Superintelligence?
52:08 Exciting Directions for AI
