Uploaded September 2026 | Updated November 2026, 4 days ago
Frank Hutter, co-founder of Prior Labs, talks about TabPFN, a tabular foundation model that makes predictions in a single forward pass, and the research behind it.
TabPFN is pre-trained on synthetic datasets drawn from a prior over structural causal models, rather than on real data. At prediction time it takes the whole training table as context and outputs an approximation of the Bayesian posterior predictive distribution, without per-dataset training or hyperparameter search. Frank explains how this grew out of his earlier work on AutoML and neural architecture search, how the priors are built and revised, and why tabular data was hard for deep learning for so long.
The conversation also covers the TabArena benchmark, how the architecture changed from TabPFN v1 to v3, scaling to larger tables, using the model with coding agents, test-time compute, Google's TabFM, causal inference and interventions, and relational data. At the end, a short update Frank recorded after the interview covers the TabPFN-3.5 release.
Prior Labs:
TabPFN-3.5: priorlabs.ai/tabpfn-3-5
Careers: priorlabs.ai/careers
TOC:
00:00 Introduction
00:44 Welcome and Frank's background
02:05 Why tabular data was hard for deep learning
07:50 Why there is little public tabular data
10:17 Pre-training on synthetic data
12:52 The TabArena benchmark
19:28 From AutoML to neural architecture search
26:34 TabPFN as a learned algorithm
30:50 Bayesian prediction in one forward pass
39:37 Scaling to larger tables
43:47 Why LLMs struggle with large tables
47:48 Using TabPFN with coding agents
55:15 Beyond classification and regression
57:47 Output heads and architecture from v1 to v3
1:05:29 Test-time compute and adaptation
1:12:26 Interpolation and extrapolation
1:13:32 Google's TabFM
1:16:53 How the priors are designed
1:18:40 Correlation, causation and interventions
1:29:06 Causal foundation models
1:33:21 Language models and numerical data
1:35:22 Relational and multimodal data
1:38:31 Use in organisations
1:46:38 The open research arm
1:50:21 Update: TabPFN-3.5
1:52:38 Closing
REFS:
AdamW (Loshchilov & Hutter)
arxiv.org/abs/1711.05101
SGDR (Loshchilov & Hutter)
arxiv.org/abs/1608.03983
TabNet (Arik & Pfister)
arxiv.org/abs/1908.07442
Deep Sets (Zaheer et al.)
arxiv.org/abs/1703.06114
TabPFN v2, Nature (Hollmann et al., 2025)
nature.com/articles/s41586-024-08328-6
Transformers Can Do Bayesian Inference (Müller et al.)
arxiv.org/abs/2112.10510
TabArena (Erickson et al.)
arxiv.org/abs/2506.16791
AutoGluon-Tabular (Erickson et al.)
arxiv.org/abs/2003.06505
Beyond IID: How General Are Tabular Foundation Models, Really?
arxiv.org/abs/2606.30410
Neural Architecture Search: A Survey (Elsken, Metzen & Hutter)
arxiv.org/abs/1808.05377
Auto-WEKA (Thornton et al.)
cs.ubc.ca/~hutter/papers/AutoWEKA-KDD2013.pdf
TabPFN v1 (Hollmann et al., 2022)
arxiv.org/abs/2207.01848
TabPFN-3 technical report
arxiv.org/abs/2605.13986
TabPFN-2.5 report
arxiv.org/abs/2511.08667
CAAFE (Hollmann et al.)
arxiv.org/abs/2305.03403
TabICL (Qu et al.)
arxiv.org/abs/2502.05564
TabICLv2 (Qu et al.)
arxiv.org/abs/2602.11139
TuneTables (Feuer et al.)
arxiv.org/abs/2402.11137
A Spline Theory of Deep Networks (Balestriero & Baraniuk)
https://proceedings.mlr.press/v80/balestriero18b.html
Google TabFM
https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data/
TALENT benchmark (Ye et al.)
arxiv.org/abs/2407.00956
Do-PFN (Robertson et al.)
arxiv.org/abs/2506.06039
CausalPFN (Balazadeh et al.)
arxiv.org/abs/2506.07918
Causal Foundation Models with Partial Graphs (Reuter et al.)
arxiv.org/abs/2602.14972
RelBench (Robinson et al.)
arxiv.org/abs/2407.20060
RelArena-α, TabPFN-Rel and RPI
arxiv.org/abs/2608.16319
TabPFN on GitHub
github.com/PriorLabs/TabPFN
Prior Labs technical reports
priorlabs.ai/technical-reports
TabPFN-3.5 technical report
arxiv.org/abs/2609.17895
TabPFN-3.5 model page (Prior Labs)
priorlabs.ai/tabpfn-3-5
Otto Group Product Classification Challenge (Kaggle, 2015)
kaggle.com/competitions/otto-group-product-classification-challenge
---
RESCRIPT:
app.rescript.info/share/e99676c25ee6189fbf54c9be07eb623e
Frank Hutter, co-founder of Prior Labs, talks about TabPFN, a tabular foundation model that makes predictions in a single forward pass, and the research behind it.
TabPFN is pre-trained on synthetic datasets drawn from a prior over structural causal models, rather than on real data. At prediction time it takes the whole training table as context and outputs an approximation of the Bayesian posterior predictive distribution, without per-dataset training or hyperparameter search. Frank explains how this grew out of his earlier work on AutoML and neural architecture search, how the priors are built and revised, and why tabular data was hard for deep learning for so long.
The conversation also covers the TabArena benchmark, how the architecture changed from TabPFN v1 to v3, scaling to larger tables, using the model with coding agents, test-time compute, Google's TabFM, causal inference and interventions, and relational data. At the end, a short update Frank recorded after the interview covers the TabPFN-3.5 release.
Prior Labs:
TabPFN-3.5: priorlabs.ai/tabpfn-3-5
Careers: priorlabs.ai/careers
TOC:
00:00 Introduction
00:44 Welcome and Frank's background
02:05 Why tabular data was hard for deep learning
07:50 Why there is little public tabular data
10:17 Pre-training on synthetic data
12:52 The TabArena benchmark
19:28 From AutoML to neural architecture search
26:34 TabPFN as a learned algorithm
30:50 Bayesian prediction in one forward pass
39:37 Scaling to larger tables
43:47 Why LLMs struggle with large tables
47:48 Using TabPFN with coding agents
55:15 Beyond classification and regression
57:47 Output heads and architecture from v1 to v3
1:05:29 Test-time compute and adaptation
1:12:26 Interpolation and extrapolation
1:13:32 Google's TabFM
1:16:53 How the priors are designed
1:18:40 Correlation, causation and interventions
1:29:06 Causal foundation models
1:33:21 Language models and numerical data
1:35:22 Relational and multimodal data
1:38:31 Use in organisations
1:46:38 The open research arm
1:50:21 Update: TabPFN-3.5
1:52:38 Closing
REFS:
AdamW (Loshchilov & Hutter)
arxiv.org/abs/1711.05101
SGDR (Loshchilov & Hutter)
arxiv.org/abs/1608.03983
TabNet (Arik & Pfister)
arxiv.org/abs/1908.07442
Deep Sets (Zaheer et al.)
arxiv.org/abs/1703.06114
TabPFN v2, Nature (Hollmann et al., 2025)
nature.com/articles/s41586-024-08328-6
Transformers Can Do Bayesian Inference (Müller et al.)
arxiv.org/abs/2112.10510
TabArena (Erickson et al.)
arxiv.org/abs/2506.16791
AutoGluon-Tabular (Erickson et al.)
arxiv.org/abs/2003.06505
Beyond IID: How General Are Tabular Foundation Models, Really?
arxiv.org/abs/2606.30410
Neural Architecture Search: A Survey (Elsken, Metzen & Hutter)
arxiv.org/abs/1808.05377
Auto-WEKA (Thornton et al.)
cs.ubc.ca/~hutter/papers/AutoWEKA-KDD2013.pdf
TabPFN v1 (Hollmann et al., 2022)
arxiv.org/abs/2207.01848
TabPFN-3 technical report
arxiv.org/abs/2605.13986
TabPFN-2.5 report
arxiv.org/abs/2511.08667
CAAFE (Hollmann et al.)
arxiv.org/abs/2305.03403
TabICL (Qu et al.)
arxiv.org/abs/2502.05564
TabICLv2 (Qu et al.)
arxiv.org/abs/2602.11139
TuneTables (Feuer et al.)
arxiv.org/abs/2402.11137
A Spline Theory of Deep Networks (Balestriero & Baraniuk)
https://proceedings.mlr.press/v80/balestriero18b.html
Google TabFM
https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data/
TALENT benchmark (Ye et al.)
arxiv.org/abs/2407.00956
Do-PFN (Robertson et al.)
arxiv.org/abs/2506.06039
CausalPFN (Balazadeh et al.)
arxiv.org/abs/2506.07918
Causal Foundation Models with Partial Graphs (Reuter et al.)
arxiv.org/abs/2602.14972
RelBench (Robinson et al.)
arxiv.org/abs/2407.20060
RelArena-α, TabPFN-Rel and RPI
arxiv.org/abs/2608.16319
TabPFN on GitHub
github.com/PriorLabs/TabPFN
Prior Labs technical reports
priorlabs.ai/technical-reports
TabPFN-3.5 technical report
arxiv.org/abs/2609.17895
TabPFN-3.5 model page (Prior Labs)
priorlabs.ai/tabpfn-3-5
Otto Group Product Classification Challenge (Kaggle, 2015)
kaggle.com/competitions/otto-group-product-classification-challenge
---
RESCRIPT:
app.rescript.info/share/e99676c25ee6189fbf54c9be07eb623e

![LSTM: The Comeback Story? [Prof. Sepp Hochreiter]
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!
https://centml.ai/pricing/
Sepp Hochreiter, the inventor of LSTM networks, makes a forceful case that large language models are fundamentally limited they store and retrieve human knowledge rather than reason about it. He walks through the technical history from the original 1991 vanishing gradient problem through to his latest work on xLSTM, explaining how exponential gating and matrix memory replace the old sigmoid bottleneck to create architectures that can compete with Transformers on sequence modeling while maintaining fixed memory footprints.
The conversation covers the practical advantages of xLSTM for industrial applications like robotics simulation and discrete element modeling, where constant memory consumption matters more than raw benchmark performance. Hochreiter also discusses the neuro-symbolic Pi AI project at JKU Linz, his view that genuine reasoning requires something beyond pattern matching on training data, and why he believes the current scaling paradigm will hit a wall. The technical deep-dive into FlashAttention comparisons, recurrent vs attention-based architectures, and the evolution from sigmoid to exponential gating mechanisms gives a clear picture of where LSTM-style architectures fit in the current landscape.
REFERENCES:
Paper:
[00:00:13] Long Short-Term Memory (Original LSTM Paper)
https://direct.mit.edu/neco/article-abstract/9/8/1735/6109/Long-Short-Term-Memory
[00:04:18] Kolmogorov Complexity
https://link.springer.com/article/10.1007/BF02478259
[00:19:38] xLSTM: Extended Long Short-Term Memory
https://arxiv.org/abs/2405.04517
[00:22:53] FlashAttention: Fast and Memory-Efficient Attention
https://arxiv.org/abs/2205.14135
[00:36:08] Mamba: Linear-Time Sequence Modeling
https://arxiv.org/abs/2312.00752
[00:55:03] Core Knowledge Theory of Human Cognition
https://www.harvardlds.org/wp-content/uploads/2017/01/SpelkeKinzler07-1.pdf
Book:
[00:53:18] Thinking, Fast and Slow
https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555
Company:
[01:00:18] NXAI: Industrial AI with xLSTM
https://investinaustria.at/en/blog/nxai-top-researchers-in-austria-develop-solutions-for-industrial-companies/
LINKS:
Full Transcript: https://app.rescript.info/share/50e3f79d116b1a92aa40819b6cb40eb2
Download PDF transcript: https://app.rescript.info/api/public/sessions/bdb8e291a0e4514d/pdf
Prof. Sepp Hochreiter
https://www.nx-ai.com/
https://x.com/hochreitersepp
https://scholar.google.at/citations?user=tvUH3WMAAAAJ&hl=en LSTM: The Comeback Story? [Prof. Sepp Hochreiter]](https://i.ytimg.com/vi/8u2pW2zZLCs/mqdefault.jpg)
![New Research: LMArena is rigged
We expose the serious problems with Chatbot Arena, the AI industrys most influential leaderboard. We showcase the recent paper The Leaderboard Illusion which shows that its actually being gamed by the biggest players in tech.
We comment on the recent revelation that Mark Zuckerberg openly admitted that Meta manipulated the arena by fine-tuning models specifically to top the charts. The paper uncovers how select companies receive preferential treatment, including extensive private testing and the ability to retract underperforming models. These companies also get a massive data advantage, as their models are sampled far more often, allowing them to train on arena data to boost their scores.
[HEDGING]
Metas actions were actually useful to expose a serious flaw in the arena and Zuck 100% owned it, and was super transparent about it afterwards.
SPONSOR MESSAGES:
Prolific: Quality data. From real people. For faster breakthroughs.
https://prolific.com/mlst?utm_campaign=98404559-MLST&utm_source=youtube&utm_medium=podcast&utm_content=script-4
Tufa AI Labs are hiring for ML Engineers and a Chief Scientist in Zurich/SF
https://tufalabs.ai/
Interactive Transcript:
https://app.rescript.info/public/share/I8k9saMwSQbkpL7rbZRiWCKnFHzgSLBi2atvnVACVm8
TOC:
00:00:00 - The Serious Problem with Chatbot Arena
00:01:45 - Mark Zuckerberg Admits to Hacking the Leaderboard
00:03:26 - Goodharts Law: Why the Benchmark is Broken
00:04:00 - Andrej Karpathy on Judging Subtle Model Improvements
00:06:45 - The Origin Story of Chatbot Arena
00:10:00 - How ELO Ratings Work (and Their Flaws)
00:15:15 - The Explosive Leaderboard Illusion Paper by Cohere
00:16:45 - Finding #1: Preferential Private Testing
00:18:00 - Finding #2: Unfair Data Advantage for Big Tech
00:19:15 - Finding #3: Massive Gains from Fine-Tuning on Arena Data
00:20:45 - Why Some Models Are Sampled Overwhelmingly More Than Others
00:22:15 - Coheres Proposed Solutions to Fix the Arena
00:23:45 - Analysis: How Repetitive Are User Prompts?
00:25:56 - Final Thoughts & The Arenas Controversial Response
REFS:
The Leaderboard Illusion [2025]
Shivalika Singh, Yiyang Nan, Alex Wang, Daniel DSouza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, Sara Hooker
https://arxiv.org/abs/2504.20879
LM Arena response:
https://blog.lmarena.ai/blog/2025/our-response/#:~:text=Published&text=Recently%2C%20a%20writeup%20titled%20%E2%80%9CThe,ongoing%20discussions%20with%20the%20authors.
Elo Uncovered: Robustness and Best Practices in Language Model Evaluation [2023]
https://arxiv.org/pdf/2311.17295
Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, Marzieh Fadaee
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference [2024]
https://arxiv.org/abs/2403.04132
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, Ion Stoica
Too much efficiency makes everything worse: overfitting and the strong version of Goodharts law
https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.html
Jascha Sohl-Dickstein
Primary/Senior Authors
https://x.com/singhshiviii
https://x.com/mziizm
https://x.com/sarahookr New Research: LMArena is rigged](https://i.ytimg.com/vi/9W_OhS38rIE/mqdefault.jpg)
![AutoGrad Changed Everything (Not Transformers) [Dr. Jeff Beck]
Dr. Jeff Beck, mathematician turned computational neuroscientist, joins us for a fascinating deep dive into why the future of AI might look less like ChatGPT and more like your own brain.
**SPONSOR MESSAGES START**
—
Prolific - Quality data. From real people. For faster breakthroughs.
https://www.prolific.com/?utm_source=mlst
—
**END**
*What if the key to building truly intelligent machines isnt bigger models, but smarter ones?*
In this conversation, Jeff makes a compelling case that weve been building AI backwards. While the tech industry races to scale up transformers and language models, Jeff argues were missing something fundamental: the brain doesnt work like a giant prediction engine. It works like a scientist, constantly testing hypotheses about a world made of *objects* that interact through *forces* — not pixels and tokens.
*The Bayesian Brain* — Jeff explains how your brain is essentially running the scientific method on autopilot. When you combine what you see with what you hear, youre doing optimal Bayesian inference without even knowing it. This isnt just philosophy — its backed by decades of behavioral experiments showing humans are surprisingly efficient at handling uncertainty.
*AutoGrad Changed Everything* — Forget transformers for a moment. Jeff argues the real hero of the AI boom was automatic differentiation, which turned AI from a math problem into an engineering problem. But in the process, we lost sight of what actually makes intelligence work.
*The Cat in the Warehouse Problem* — Heres where it gets practical. Imagine a warehouse robot thats never seen a cat. Current AI would either crash or make something up. Jeffs approach? Build models that *know what they dont know*, can phone a friend to download new object models on the fly, and keep learning continuously. Its like giving robots the ability to say wait, what IS that? instead of confidently being wrong.
*Why Language is a Terrible Model for Thought* — In a provocative twist, Jeff argues that grounding AI in language (like we do with LLMs) is fundamentally misguided. Self-report is the least reliable data in psychology — people routinely explain their own behavior incorrectly. We should be grounding AI in physics, not words.
*The Future is Lots of Little Models* — Instead of one massive neural network, Jeff envisions AI systems built like video game engines: thousands of small, modular object models that can be combined, swapped, and updated independently. Its more efficient, more flexible, and much closer to how we actually think.
Whether youre an AI researcher, a robotics enthusiast, or just curious about how minds — biological or artificial — actually work, this conversation offers a refreshingly different perspective on where intelligence comes from and where its going.
Rescript: https://app.rescript.info/public/share/D-b494t8DIV-KRGYONJghvg-aelMmxSDjKthjGdYqsE
TIMESTAMPS:
00:00:00 Introduction & The Bayesian Brain
00:01:25 Bayesian Inference & Information Processing
00:05:17 The Brain Metaphor: From Levers to Computers
00:10:13 Micro vs. Macro Causation & Instrumentalism
00:16:59 The Active Inference Community & AutoGrad
00:22:54 Object-Centered Models & The Grounding Problem
00:35:50 Scaling Bayesian Inference & Architecture Design
00:48:05 The Cat in the Warehouse: Solving Generalization
00:58:17 Alignment via Belief Exchange
01:05:24 Deception, Emergence & Cellular Automata
REFERENCES:
Paper:
[00:00:24] Zoubin Ghahramani (Google DeepMind)
https://pmc.ncbi.nlm.nih.gov/articles/PMC3538441/pdf/rsta201
[00:19:20] Mamba: Linear-Time Sequence Modeling
https://arxiv.org/abs/2312.00752
[00:27:36] xLSTM: Extended Long Short-Term Memory
https://arxiv.org/abs/2405.04517
[00:41:12] 3D Gaussian Splatting
https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
[01:07:09] Lenia: Biology of Artificial Life
https://arxiv.org/abs/1812.05433
[01:08:20] Growing Neural Cellular Automata
https://distill.pub/2020/growing-ca/
[01:14:05] DreamCoder
https://arxiv.org/abs/2006.08381
[01:14:58] The Genomic Bottleneck
https://www.nature.com/articles/s41467-019-11786-6
Person:
[00:16:42] Karl Friston (UCL)
https://www.youtube.com/watch?v=PNYWi996Beg AutoGrad Changed Everything (Not Transformers) [Dr. Jeff Beck]](https://i.ytimg.com/vi/9suqiofCiwM/mqdefault.jpg)
![Intelligence is collective, not artificial — Prof. Michael I. Jordan (UC Berkeley / Inria)
Michael I. Jordan, described by Science magazine as the most influential computer scientist alive, has never thought of himself as an AI researcher. In this conversation he explains why that distinction matters.
SPONSOR:
Cyber Fund built the Monastery to help founders ship products that were impossible a year ago. Applications for Batch 1 are now open.
Apply now: https://cyber.fund
Jordan trained as a statistician and cognitive scientist, and his career has been spent building machine learning systems that work in the real world: supply chains, commerce, healthcare, and large economic systems. When the field rebranded itself as AI and then AGI, he did not follow. Instead he argues that the framing is wrong. AI is better understood as a collective economic system than as a race to build a disembodied superintelligence.
We talk about why AGI is mostly a PR term, what machine learning achieved before the LLM hype cycle, and why the assistant-on-your-shoulder vision may be less compelling than it sounds. Jordan explains why explanations need to be actionable, not merely mechanistic; why AlphaFolds missing error bars matter; how prediction-powered inference changes the picture; and why drug discovery is an incentive-design problem rather than a pure pattern-matching problem.
ERRATA: Science magazine ranked him the most influential computer scientist, not Nature
TIMESTAMPS:
00:00:00 Cold open: A demoralizing message to young builders
00:02:04 CyberFund sponsor read
00:02:50 From symbolic AI to machine learning systems
00:05:42 Why AGI is mostly a PR term
00:08:48 A collectivist, economic perspective on AI
00:11:33 Why LLMs need system design, not hype
00:14:50 Predictability beats faux understanding
00:17:55 AlphaFold, bias, and prediction-powered inference
00:21:48 Stop anthropomorphizing intelligence
00:27:44 Drug discovery as an incentive problem
00:32:29 The three-layer data market
00:38:07 Social knowledge, markets, and culture
00:45:39 Creator economics beyond Spotify
00:48:30 How science-fiction AI narratives mislead young builders
00:51:45 AI should improve humans, not replace them
00:56:42 Safety is a property of the whole system
00:58:12 Silicon Valley gurus and the cream off the top
01:00:47 Game theory, mechanism design, and contracts
01:04:39 Conformal prediction, e-values, and anytime inference
01:08:11 A new liberal arts triangle for the AI era
01:11:30 The Bayesian duck and markets as uncertainty reduction
REFERENCES:
person:
[00:02:50] Michael I. Jordan (homepage)
https://people.eecs.berkeley.edu/~jordan/
paper:
[00:06:01] A Collectivist, Economic Perspective on AI
https://arxiv.org/abs/2507.06268
[00:18:09] AlphaFold
https://www.nature.com/articles/s41586-021-03819-2
[00:20:36] Prediction-Powered Inference
https://arxiv.org/abs/2301.09633
[00:24:38] On the Measure of Intelligence
https://arxiv.org/abs/1911.01547
[00:33:47] On Three-Layer Data Markets
https://arxiv.org/abs/2402.09697
[01:04:39] Conformal Prediction with Conditional Guarantees
https://arxiv.org/abs/2107.07511
[01:04:51] A Tutorial on Conformal Prediction
https://www.jmlr.org/papers/v9/shafer08a.html
[01:06:00] E-Values Expand the Scope of Conformal Prediction
https://arxiv.org/abs/2503.13050
[01:08:23] Computational Thinking
https://www.cs.cmu.edu/~CompThink/papers/Wing06.pdf
other:
[00:11:33] The Bitter Lesson
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
[00:20:50] How to use AI for discovery without leading science astray
https://news.berkeley.edu/2023/11/09/how-to-use-ai-for-discovery-without-leading-science-astray/
[00:28:20] How Should the FDA Test?
https://rdi.berkeley.edu/events/sbc-assets/pdfs/Summit%20session%20speaker%20slides%20submission%20form-s1-5%20%28File%20responses%29/Slides%20in%20PDF%20%28Please%20name%20the%20submitted%20file%20as%20_firstname lastname slides.pdf%29.%20%28File%20responses%29/27-Michael%20Jordan-Session%20V.pdf#page=15
[00:28:40] Michael I. Jordan Session V Slides
https://rdi.berkeley.edu/events/sbc-assets/pdfs/Summit%20session%20speaker%20slides%20submission%20form-s1-5%20%28File%20responses%29/Slides%20in%20PDF%20%28Please%20name%20the%20submitted%20file%20as%20_firstname lastname slides.pdf%29.%20%28File%20responses%29/27-Michael%20Jordan-Session%20V.pdf#page=1
[01:08:13] Three Foundational Disciplines
https://rdi.berkeley.edu/events/sbc-assets/pdfs/Summit%20session%20speaker%20slides%20submission%20form-s1-5%20%28File%20responses%29/Slides%20in%20PDF%20%28Please%20name%20the%20submitted%20file%20as%20_firstname lastname slides.pdf%29.%20%28File%20responses%29/27-Michael%20Jordan-Session%20V.pdf#page=35
organization:
[00:45:58] UnitedMasters
https://www.unitedmasters.com/
book:
[00:48:30] Human Compatible: Artificial Intelligence and the Problem of Control
https://aima.cs.berkeley.edu/~russell/hc.html
[01:00:56] Theory of Games and Economic Behavior
https://press.princeton.edu/books/paperback/9780691130613/theory-of-games-and-economic-behavior Intelligence is collective, not artificial — Prof. Michael I. Jordan (UC Berkeley / Inria)](https://i.ytimg.com/vi/AREWYbVtX64/mqdefault.jpg)
![The Final Boss of Deep Learning
We often think of Large Language Models (LLMs) as all-knowing, but as the team reveals, they still struggle with the logic of a second-grader. Why can’t ChatGPT reliably add large numbers? Why does it hallucinate the laws of physics? The answer lies in the architecture. This episode explores how *Category Theory*—an ultra-abstract branch of mathematics—could provide the Periodic Table for neural networks, turning the alchemy of modern AI into a rigorous science.
In this deep-dive exploration, *Andrew Dudzik*, *Petar Velichkovich*, *Taco Cohen*, *Bruno Gavranović*, and *Paul Lessard* join host *Tim Scarfe* to discuss the fundamental limitations of today’s AI and the radical mathematical framework that might fix them.
Key Insights in This Episode:
* *The Addition Problem:* *Andrew Dudzik* explains why LLMs dont actually know math—they just recognize patterns. When you change a single digit in a long string of numbers, the pattern breaks because the model lacks the internal machinery to perform a simple carry operation.
* *Beyond Alchemy:* *Tim Scarfe* argues that deep learning is currently in its alchemy phase—we have powerful results, but we lack a unifying theory. Category Theory is proposed as the framework to move AI from trial-and-error to principled engineering. [00:13:49]
* *Algebra with Colors:* To make Category Theory accessible, the guests use brilliant analogies—like thinking of matrices as *magnets with colors* that only snap together when the types match. This partial compositionality is the secret to building more complex internal reasoning. [00:09:17]
* *Synthetic vs. Analytic Math:* *Paul Lessard* breaks down the philosophical shift needed in AI research: moving from Analytic math (what things are made of) to Synthetic math (how things behave and relate to one another). [00:23:41]
* *The 4D Carry:* In a mind-blowing conclusion, the team discusses how simple algorithmic tasks, like carrying the one in addition, actually relate to complex geometric structures like *Hopf Fibrations*. [00:39:30]
Why This Matters for AGI
If we want AI to solve the worlds hardest scientific problems, it cant just be a stochastic parrot. It needs to internalize the rules of logic and computation. By imbuing neural networks with categorical priors, researchers are attempting to build a future where AI doesnt just predict the next word—it understands the underlying structure of the universe.
TIMESTAMPS:
00:00:00 The Failure of LLM Addition & Physics
00:01:26 Tool Use vs Intrinsic Model Quality
00:03:07 Efficiency Gains via Internalization
00:04:28 Geometric Deep Learning & Equivariance
00:07:05 Limitations of Group Theory
00:09:17 Category Theory: Algebra with Colors
00:11:25 The Systematic Guide of Lego-like Math
00:13:49 The Alchemy Analogy & Unifying Theory
00:15:33 Information Destruction & Reasoning
00:18:00 Pathfinding & Monoids in Computation
00:20:15 System 2 Reasoning & Error Awareness
00:23:31 Analytic vs Synthetic Mathematics
00:25:52 Morphisms & Weight Tying Basics
00:26:48 2-Categories & Weight Sharing Theory
00:28:55 Higher Categories & Emergence
00:31:41 Compositionality & Recursive Folds
00:34:05 Syntax vs Semantics in Network Design
00:36:14 Homomorphisms & Multi-Sorted Syntax
00:39:30 The Carrying Problem & Hopf Fibrations
REFERENCES:
Company:
Model:
[00:01:05] Veo
https://deepmind.google/models/veo/
[00:01:10] Genie
https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/
Paper:
[00:04:30] Geometric Deep Learning Blueprint
https://arxiv.org/abs/2104.13478
[00:16:45] AlphaGeometry
https://deepmind.google/blog/alphageometry-an-olympiad-level-ai-system-for-geometry/
[00:16:55] AlphaCode
https://arxiv.org/abs/2203.07814
[00:17:05] FunSearch
https://www.nature.com/articles/s41586-023-06924-6
[00:37:00] Attention Is All You Need
https://arxiv.org/abs/1706.03762
[00:43:00] Categorical Deep Learning
https://arxiv.org/abs/2402.15332
Thanks to ly3xqhl8g9 on our Discord server for the draft show review! The Final Boss of Deep Learning](https://i.ytimg.com/vi/AWqvBdqCAAE/mqdefault.jpg)

![AI in 2024 - efficiency over model size (Nick Jakobi)
This video is sponsored by Cohere
Nick Jakobi, Director of Product for the modeling team at Cohere, discusses the practical realities of deploying large language models in enterprise environments. The conversation covers how Cohere differentiates in a crowded LLM market through enterprise-focused capabilities like retrieval-augmented generation (RAG), tool use, and multilingual support, rather than competing purely on model scale.
Jakobi offers a candid perspective on the economics of model selection: most enterprise customers gravitate toward the smallest model that solves their use case, making cost and latency as important as raw capability. He describes the emerging token game where model providers compete on price-performance ratios, and argues that making medium-sized models cheaper and faster may be more commercially valuable than pushing the frontier of model intelligence.
The discussion extends into broader questions about how generative AI is reshaping education, workforce dynamics, and information ecosystems. Jakobi shares personal reflections on whether traditional education remains relevant when AI tools are becoming ubiquitous, and offers a measured take on AI safety that centers on the practical challenges of disinformation and autonomous agency rather than speculative existential risks.
REFERENCES:
Company:
[00:00:00] Cohere
https://cohere.com/
Paper:
[00:00:00] Episode Shownotes
https://www.dropbox.com/scl/fi/sjwqllqh1ughuu8ismonk/NickJakobi2.pdf?rlkey=d20wydhhala2hmh4yaoo55wyy&st=unso2w6i&dl=0
[00:00:30] RAG Paper
https://arxiv.org/abs/2005.11401
Person:
[00:00:20] Nick Jakobi
https://www.linkedin.com/in/nickjakobi/
Product:
[00:14:50] Cohere Command R
https://cohere.com/command
Tool:
[00:20:50] LMSYS Chatbot Arena
https://chat.lmsys.org/
LINKS:
Full Transcript: https://app.rescript.info/share/d694e87868aa6d0272fe26e0a7b4de87
Download PDF transcript: https://app.rescript.info/api/public/sessions/72c99a6b30e9b8d0/pdf AI in 2024 - efficiency over model size (Nick Jakobi)](https://i.ytimg.com/vi/BJMi_dAwE8k/mqdefault.jpg)


![Test-Time Adaptation: A New Frontier in AI [Dr. Jonas Hübotter]
Jonas Hübotter, PhD student at ETH Zurichs Institute for Machine Learning, discusses his groundbreaking research on test-time computation and local learning. He demonstrates how smaller models can outperform larger ones by 30x through strategic test-time computation and introduces a novel paradigm combining inductive and transductive learning approaches.
Using Bayesian linear regression as a surrogate model for uncertainty estimation, Jonas explains how models can efficiently adapt to specific tasks without massive pre-training. He draws an analogy to Google Earths variable resolution system to illustrate dynamic resource allocation based on task complexity.
The conversation explores the future of AI architecture, envisioning systems that continuously learn and adapt beyond current monolithic models. Jonas concludes by proposing hybrid deployment strategies combining local and cloud computation, suggesting a future where compute resources are allocated based on task complexity rather than fixed model size.
This research represents a significant shift in machine learning, prioritizing intelligent resource allocation and adaptive learning over traditional scaling approaches.
SPONSOR MESSAGES:
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.
https://centml.ai/pricing/
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on ARC and AGI, they just acquired MindsAI - the current winners of the ARC challenge. Are you interested in working on ARC, or getting involved in their events? Goto https://tufalabs.ai/
Transcription, references and show notes PDF download:
https://www.dropbox.com/scl/fi/cxg80p388snwt6qbp4m52/JonasFinal.pdf?rlkey=glk9mhpzjvesanlc14rtpvk4r&st=6qwi8n3x&dl=0
Jonas Hübotter
https://jonhue.github.io/
https://scholar.google.com/citations?user=pxi_RkwAAAAJ
Transductive Active Learning: Theory and Applications (NeurIPS 2024)
https://arxiv.org/pdf/2402.15898
EFFICIENTLY LEARNING AT TEST-TIME: ACTIVE FINE-TUNING OF LLMS (SIFT)
https://arxiv.org/pdf/2410.08020
TOC:
1. Test-Time Computation Fundamentals
[00:00:00] Intro
[00:03:10] 1.1 Test-Time Computation and Model Performance Comparison
[00:05:52] 1.2 Retrieval Augmentation and Machine Teaching Strategies
[00:09:40] 1.3 In-Context Learning vs Fine-Tuning Trade-offs
2. System Architecture and Intelligence
[00:15:58] 2.1 System Architecture and Intelligence Emergence
[00:23:22] 2.2 Active Inference and Constrained Agency in AI
[00:29:52] 2.3 Evolution of Local Learning Methods
[00:32:05] 2.4 Vapniks Contributions to Transductive Learning
3. Resource Optimization and Local Learning
[00:34:35] 3.1 Computational Resource Allocation in ML Models
[00:35:30] 3.2 Historical Context and Traditional ML Optimization
[00:37:55] 3.3 Variable Resolution Processing and Active Inference in ML
[00:43:01] 3.4 Local Learning and Base Model Capacity Trade-offs
[00:48:04] 3.5 Active Learning vs Local Learning Approaches
4. Information Retrieval and Model Interpretability
[00:51:08] 4.1 Information Retrieval and Nearest Neighbor Limitations
[01:03:07] 4.2 Model Interpretability and Surrogate Models
[01:15:03] 4.3 Bayesian Uncertainty Estimation and Surrogate Models
5. Distributed Systems and Deployment
[01:23:56] 5.1 Memory Architecture and Controller Systems
[01:28:14] 5.2 Evolution from Static to Distributed Learning Systems
[01:38:03] 5.3 Transductive Learning and Model Specialization
[01:41:58] 5.4 Hybrid Local-Cloud Deployment Strategies Test-Time Adaptation: A New Frontier in AI [Dr. Jonas Hübotter]](https://i.ytimg.com/vi/C6sSs6NgANo/mqdefault.jpg)