Uploaded June 2026 | Updated November 2026, 17 hours ago
[2026 - DAY 2 - ANALYTICS & DATA SCI] Education researchers have spent a century figuring out how to build environments where complex, unpredictable systems self-correct through structured feedback. Software engineers building AI agents are solving the exact same problem from scratch and ignoring all of it. This talk bridges that gap. I'm a former classroom teacher turned AI engineer, and I'll walk you through five pedagogical frameworks that map directly to eval design patterns for AI agents: backward design (define success criteria before you build), formative assessment (eval continuously, not just at the end), rubric design (multi-dimensional scoring instead of pass/fail), error analysis (categorize failure modes because same symptom doesn't mean same cause), and differentiated feedback (the agent, the user, and the knowledge base each need their own signal channel). What ties them together is back pressure: each framework is a way to capture signal from problems and route it to where it drives change. That's what makes a system self-correcting instead of just self-reporting. Most AI observability is still about watching systems after the fact. This talk is about designing systems where the eval layer captures back pressure from failures and feeds it back upstream, so the system iterates on itself. I've built production agents and won hackathons with this approach, and the core insight is simple: the best eval systems aren't tests, they're environments. And nobody knows more about designing those environments than teachers.
SPEAKER:
Andrew Zigler - GTM Engineer, LinearB & Podcast Host, Dev Interrupted
👉 Sign up for our "No BS" Newsletter to get the latest technical data & AI content: aicouncil.com/newsletter
ABOUT AI COUNCIL:
AI Council brings together the brightest minds in data to share industry knowledge, technical architectures and best practices in building cutting edge data & AI systems and tools.
FIND US:
Website: aicouncil.com
LinkedIn: linkedin.com/company/aicouncilconf
X: https://x.com/aicouncilconf
[2026 - DAY 2 - ANALYTICS & DATA SCI] Education researchers have spent a century figuring out how to build environments where complex, unpredictable systems self-correct through structured feedback. Software engineers building AI agents are solving the exact same problem from scratch and ignoring all of it. This talk bridges that gap. I'm a former classroom teacher turned AI engineer, and I'll walk you through five pedagogical frameworks that map directly to eval design patterns for AI agents: backward design (define success criteria before you build), formative assessment (eval continuously, not just at the end), rubric design (multi-dimensional scoring instead of pass/fail), error analysis (categorize failure modes because same symptom doesn't mean same cause), and differentiated feedback (the agent, the user, and the knowledge base each need their own signal channel). What ties them together is back pressure: each framework is a way to capture signal from problems and route it to where it drives change. That's what makes a system self-correcting instead of just self-reporting. Most AI observability is still about watching systems after the fact. This talk is about designing systems where the eval layer captures back pressure from failures and feeds it back upstream, so the system iterates on itself. I've built production agents and won hackathons with this approach, and the core insight is simple: the best eval systems aren't tests, they're environments. And nobody knows more about designing those environments than teachers.
SPEAKER:
Andrew Zigler - GTM Engineer, LinearB & Podcast Host, Dev Interrupted
👉 Sign up for our "No BS" Newsletter to get the latest technical data & AI content: aicouncil.com/newsletter
ABOUT AI COUNCIL:
AI Council brings together the brightest minds in data to share industry knowledge, technical architectures and best practices in building cutting edge data & AI systems and tools.
FIND US:
Website: aicouncil.com
LinkedIn: linkedin.com/company/aicouncilconf
X: https://x.com/aicouncilconf
![From Scaling to Observability Solving Key Challenges for Distributed ML with Ray
[2025 - Day 2 - MLOps & Platforms] Nikita Vemuri shares insights from scaling distributed ML training across thousands of nodes with Ray, exploring the observability challenges of tracking vast amounts of system data in multi-node environments. Whether youre running large-scale ML workloads or building monitoring systems for distributed training, this talk offers valuable perspectives on correlating information across clusters and designing observability stacks that surface relevant insights while maintaining data privacy.
ABOUT THE SPEAKER:
Nikita Vemuri, Software Engineer, Anyscale -
🎟️ GET YOUR TICKET TO AI COUNCIL 2026 🎟️
Meet the worlds top AI infrastructure minds where architects of AI share what works. Three days of high-quality technical talks and meaningful interactions.
→ https://aicouncil.com/sf-2026
⚡ FIND US:
X: https://x.com/AICouncilConf
LinkedIn: https://www.linkedin.com/company/aicouncilconf/
Website: https://aicouncil.com/ From Scaling to Observability Solving Key Challenges for Distributed ML with Ray](https://i.ytimg.com/vi/nfrmfyoKs6A/mqdefault.jpg)
![Jev: ChatGPT Co-Creator’s Answer to RLHF
4 months before TypeSafe AI launched Jev, founder Diogo Almeida stood up at AI Council and laid out the problem Jev was built to solve.
He never names the product in this talk. He says only that TypeSafe is going all in on automation and getting ready to ship in a couple of months, then spends 36 minutes on why he thinks the rest of the field is pointed the wrong way. TypeSafe came out of stealth on September 15, 2026 with $40M led by DCVC and launched Jev, a model that outputs typed decisions with probabilities for software to act on.
Almeida spent four and a half years at OpenAI, where he co-authored the InstructGPT paper and the GPT-4 Technical Report and worked on the RLHF research behind ChatGPT. His argument here starts from that work. His read is that roughly all production LLMs are trained with RLHF or some variation of it, and RLHF optimizes for human preference. Human preference is a different target from doing the task correctly. That gap is why AI looks superhuman on assistance work and falls apart on automation work that looks easier.
He walks through mode collapse, Yann LeCuns doomed-LLM argument, why a 1.3B-parameter InstructGPT model was preferred to 175B GPT-3 at following instructions, and where he thinks Suttons bitter lesson is incomplete. He closes on the question that started TypeSafe, which is what it would take to treat language models as a primitive that software calls, the way it calls a database or an API.
Recorded live at AI Council 2026 in San Francisco, May 12-14.
SPEAKER
Diogo Almeida, Co-founder and CEO, TypeSafe AI, InstructGPT and GPT-4 co-author
X: https://x.com/CompleteSkeptic
LinkedIn: https://www.linkedin.com/in/diogomda/
TypeSafe AI: https://typesafe.ai/
Jev: https://typesafe.ai/blog/introducing-system-one-models-and-jev
CHAPTERS
0:00 AI is too good to be true and too bad to be useful
0:51 Four and a half years at OpenAI
2:11 Where is the economic revolution?
5:23 Assistance work versus automation work
6:09 But arent coding agents automation?
9:43 When is a result too good to be true?
11:18 Suttons bitter lesson, and the bitterest lesson
13:49 What RLHF is and what it costs you
14:44 Yann LeCuns doomed-LLM argument
15:49 Mode collapse, RLHFs deal with the devil
18:18 Is software engineering easier than drive-thrus?
19:49 Why a model cant be trusted with stakes
20:42 Is ChatGPT cooked?
25:33 Type-safe language models, going beyond strings
27:13 The FLAN lesson, when the whole field is wrong
29:40 Can one model do assistance and automation?
31:08 What TypeSafe is building
33:42 Summary
35:00 Was RLHF AGI, and what was missing
[2026 - DAY 1 - INFERENCE SYSTEMS]
Sign up for our No BS Newsletter to get the latest technical data & AI content: https://aicouncil.com/newsletter
ABOUT AI COUNCIL:
AI Council brings together the brightest minds in data to share industry knowledge, technical architectures and best practices in building cutting edge data & AI systems and tools.
FIND US:
Website: https://aicouncil.com/
LinkedIn: https://www.linkedin.com/company/aicouncilconf/
X: https://x.com/aicouncilconf Jev: ChatGPT Co-Creator’s Answer to RLHF](https://i.ytimg.com/vi/o-y1HJ6buGQ/mqdefault.jpg)
![DuckDB Co-Creator Hannes Mühleisen on Why Single-Node Beats Distributed
[2025 - Day 1 - Data Eng & Infra] Hannes Mühleisen shares insights from co-creating DuckDB, exploring how this in-process analytical engines small footprint and high per-core efficiency unlocks creative data interaction possibilities. For engineers building data pipelines or deploying analytics anywhere from servers to small devices, this talk offers valuable guidance on leveraging DuckDBs vectorized processing and larger-than-memory capabilities for previously impossible use cases.
ABOUT THE SPEAKER:
Hannes Mühleisen, Co-Creator, DuckDB
🎟️ GET YOUR TICKET TO AI COUNCIL 2026 🎟️
Meet the worlds top AI infrastructure minds where architects of AI share what works. Three days of high-quality technical talks and meaningful interactions.
→ https://aicouncil.com/sf-2026
⚡ FIND US:
X: https://x.com/AICouncilConf
LinkedIn: https://www.linkedin.com/company/aicouncilconf/
Website: https://aicouncil.com/ DuckDB Co-Creator Hannes Mühleisen on Why Single-Node Beats Distributed](https://i.ytimg.com/vi/o53onmgnQDU/mqdefault.jpg)
![Why dbt Acquired SDF Building true SQL Comprehension
[2025 - Day 1 - Data Eng & Infra] Elias DeFaria shares insights from building SDFs multi-dialect SQL compiler that resolves proprietary SQL dialects into unified logical plans, exploring the breakthrough technology now powering dbt following the acquisition. This talk offers valuable perspectives on normalizing complex dialects and unlocking developer experience improvements in data governance and cost optimization, especially relevant for teams working with cross-engine workflows or building analytics infrastructure.
ABOUT THE SPEAKER:
Elias DeFaria, Staff Product Manager, dbt | Former Co-Founder & VP Product, SDF -
🎟️ GET YOUR TICKET TO AI COUNCIL 2026 🎟️
Meet the worlds top AI infrastructure minds where architects of AI share what works. Three days of high-quality technical talks and meaningful interactions.
→ https://aicouncil.com/sf-2026
⚡ FIND US:
X: https://x.com/AICouncilConf
LinkedIn: https://www.linkedin.com/company/aicouncilconf/
Website: https://aicouncil.com/ Why dbt Acquired SDF Building true SQL Comprehension](https://i.ytimg.com/vi/oE8I2VQsKn4/mqdefault.jpg)
![AGI is Already Here (But Its Not What You Think)
[2025 - Day 3 - GenAI Applications] Joseph Gonzalez and Chenggang Wu share insights from redefining AGI through compound AI systems that compose LLMs with data and tools, exploring how orchestrating multiple models creates intelligence that rivals human capabilities. Whether youre building AI-powered workflows or developing general-purpose agents, this talk offers valuable perspectives on creating systems that outperform humans in both accuracy and efficiency while being notably faster.
ABOUT THE SPEAKERS:
Joseph Gonzalez, Professor, Head of AI, UC Berkeley / RunLLM
Chenggang Wu, Co-Founder & CTO, RunLLM -
🎟️ GET YOUR TICKET TO AI COUNCIL 2026 🎟️
Meet the worlds top AI infrastructure minds where architects of AI share what works. Three days of high-quality technical talks and meaningful interactions.
→ https://aicouncil.com/sf-2026
⚡ FIND US:
X: https://x.com/AICouncilConf
LinkedIn: https://www.linkedin.com/company/aicouncilconf/
Website: https://aicouncil.com/ AGI is Already Here (But Its Not What You Think)](https://i.ytimg.com/vi/oSKV_c2ddWA/mqdefault.jpg)
![AI Launchpad 2025: NAO
[2025 - Day 1 - AI Launchpad] Claire Gouze and Christophe Blefari share insights from Nao, an AI code editor specifically designed for data teams, exploring how to streamline fragmented analytics engineering workflows across multiple tools. This session offers valuable perspectives on creating unified environments tailored to data professionals unique modeling needs, addressing productivity challenges from constant context switching between IDEs, CLIs, data warehouse UIs, documentation sites, and BI platforms.
ABOUT THE SPEAKERS:
Claire Gouze, Co-Founder & CEO, NAO
Christophe Blefari, Co-Founder, NAO -
🎟️ GET YOUR TICKET TO AI COUNCIL 2026 🎟️
Meet the worlds top AI infrastructure minds where architects of AI share what works. Three days of high-quality technical talks and meaningful interactions.
→ https://aicouncil.com/sf-2026
⚡ FIND US:
X: https://x.com/AICouncilConf
LinkedIn: https://www.linkedin.com/company/aicouncilconf/
Website: https://aicouncil.com/ AI Launchpad 2025: NAO](https://i.ytimg.com/vi/oyJziKnpElo/mqdefault.jpg)
![Building a Data Native Agent: Cortex Code | Snowflake
[2026 - DAY 2 - WORKSHOP] We built Cortex Code to solve the context gap in AI development by engineering the industrys first data-native coding agent that is deeply coupled with the underlying data platform. We architected a system that directly interfaces with Snowflakes metadata, compute, and governance layers, giving the agent real-time environment awareness of table structures, operational semantics, and security.
In this session, we will dive into how we constructed this context-aware reasoning loop, utilized open standards like MCP and agents skills for extensibility, and solved the challenge of delivering an agent that accelerates complex data engineering and ML workflows while remaining secure-by-design. We will also share our experiences with benchmarking and optimizing Cortex Code on public and internal benchmarks.
SPEAKER:
Umesh Unnikrishnan - AI Research Lead, Snowflake
👉 Sign up for our No BS Newsletter to get the latest technical data & AI content: https://aicouncil.com/newsletter
ABOUT AI COUNCIL:
AI Council brings together the brightest minds in data to share industry knowledge, technical architectures and best practices in building cutting edge data & AI systems and tools.
FIND US:
Website: https://aicouncil.com/
LinkedIn: https://www.linkedin.com/company/aicouncilconf/
X: https://x.com/aicouncilconf Building a Data Native Agent: Cortex Code | Snowflake](https://i.ytimg.com/vi/pV3hybzf31A/mqdefault.jpg)
![Calvin French-Owen on the Future of Agentic Coding
[2026 - DAY 2 - CODING AGENTS] Coding agents are constantly changing — what works in Claude Code one week might not make sense in Codex the next. When working with coding agents, you can lean into a few core principles rooted in how they are trained and built. In this talk Calvin will share some of the high-level takeaways from building one of the first coding agents, as well as an in-the-weeds perspective on whats currently working. Calvin previously cofounded Segment and while working at OpenAI was on the team that created Codex.
SPEAKER:
Calvin French-Owen - Co-founder & CTO, Segment (acquired by Twilio)
👉 Sign up for our No BS Newsletter to get the latest technical data & AI content: https://aicouncil.com/newsletter
ABOUT AI COUNCIL:
AI Council brings together the brightest minds in data to share industry knowledge, technical architectures and best practices in building cutting edge data & AI systems and tools.
FIND US:
Website: https://aicouncil.com/
LinkedIn: https://www.linkedin.com/company/aicouncilconf/
X: https://x.com/aicouncilconf Calvin French-Owen on the Future of Agentic Coding](https://i.ytimg.com/vi/q-ntX4DLW_c/mqdefault.jpg)
![The Future of Guardrails
[2025 - Day 1 - AI Engineering] Shreya Rajpal shares insights from implementing risk management frameworks for generative AI applications, exploring comprehensive approaches to reliability and risk assessment across different AI architectures. This talk offers valuable perspectives on systematic risk management for RAG-augmented chatbots and autonomous agents, crucial for teams deploying generative AI in real-world environments while reducing potential failure modes and negative outcomes.
ABOUT THE SPEAKER:
Shreya Rajpal, Co-Founder & CEO, Guardrails -
🎟️ GET YOUR TICKET TO AI COUNCIL 2026 🎟️
Meet the worlds top AI infrastructure minds where architects of AI share what works. Three days of high-quality technical talks and meaningful interactions.
→ https://aicouncil.com/sf-2026
⚡ FIND US:
X: https://x.com/AICouncilConf
LinkedIn: https://www.linkedin.com/company/aicouncilconf/
Website: https://aicouncil.com/ The Future of Guardrails](https://i.ytimg.com/vi/qEELBE58qSI/mqdefault.jpg)

