Uploaded August 2026 | Updated September 2026, 1 week ago

---

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at notion.com/mlst

Why can deep networks discover abstractions that shallow models miss? Statistical physicist Matthieu Wyart joins Tim Scarfe to argue that the answer lies in the hidden hierarchy of data. Language and images are built from parts within parts; depth lets a network recover those coarse-grained variables and escape the curse of dimensionality.

The conversation moves from jamming transitions and rough loss surfaces to Chomsky, context-free grammars and machine creativity. Wyart explains why next-token prediction can still recover compositional structure, where current systems fall short of genuine scientific invention, and why predicting latent representations rather than raw tokens could make learning far more sample-efficient.

They also examine diffusion models, neural scaling laws and the limits of physics-inspired theory. The final question is on a personal note: if mistakes are the price of leaving the beaten path, how much scientific risk is worth taking?

---
TIMESTAMPS:
00:00:00 Can machines learn abstractions from data?
00:02:00 Notion agentic workspace
00:02:49 From statistical physics to machine learning
00:06:40 What physics can explain about learning
00:16:37 From Carnot to Chomsky bulldozer
00:21:21 How deep networks recover hidden hierarchies
00:32:43 Where machine creativity still falls short
00:40:48 How deep nets escape the curse of dimensionality
00:52:19 Why predict latents instead of tokens
01:02:49 The sample-efficiency case for latent prediction
01:08:31 Diffusion, scaling laws and text entropy
01:16:40 The scientists we learn from and the mistakes we make

---
REFERENCES:
person:
[00:00:43] Noam Chomsky
https://linguistics.mit.edu/user/chomsky/
tool:
[00:02:08] Notion Developer Platform
notion.com/en-gb/blog/introducing-developer-platform
paper:
[00:04:43] Mastering the game of Go with deep neural networks and tree search
nature.com/articles/nature16961
[00:05:52] Reconciling modern machine-learning practice and the bias-variance trade-off
arxiv.org/abs/1812.11118
[00:25:54] How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
arxiv.org/abs/2307.02129
[00:42:12] Efficient Estimation of Word Representations in Vector Space
arxiv.org/abs/1301.3781
[00:52:46] Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
arxiv.org/abs/2301.08243
[00:52:54] Learn from your own latents and not from tokens: A sample-complexity theory
arxiv.org/abs/2605.27734
[01:08:31] A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data
arxiv.org/abs/2402.16991
[01:11:39] Scaling Laws for Neural Language Models
arxiv.org/abs/2001.08361
[01:12:17] Deriving Neural Scaling Laws from the statistics of natural language
arxiv.org/abs/2602.07488
[01:13:34] Prediction and Entropy of Printed English
ieeexplore.ieee.org/document/6773263

---
LINKS:
Download PDF transcript: app.rescript.info/share/f7644cdaa86c5cc1e41e484e290f2bd4