Uploaded September 2026 | Updated November 2026, 1 day ago
Leonardo de Moura created Lean and co-created Z3.
---
This episode is sponsored by Parallel.
Parallel, where agents find answers: web search, extraction and deep research APIs built for AI agents.
Start free with the Parallel MCP server and $5 of credits every month: parallel.ai/mlst?utm_source=creator&utm_medium=podcast&utm_content=MLST
---
Tim Scarfe talks with Leo about how Lean escaped its original audience, why dependent types and Mathlib made it useful to working mathematicians, and what happens when formal verification leaves the lab. De Moura explains the small trusted kernel and independent checkers, and gives his account of the recent Collatz incident, in which a purported proof was accepted by both Lean's official kernel and Nanoda, apparently by exploiting a different bug in each.
The conversation then turns to AI-produced proofs and software. Claude agents rebuild zlib in Lean and the result is verified, yet the example exposes the loophole at the centre of the show: a proof only certifies the specification humans chose to write. De Moura describes proof search as a game, separates formal competence from mathematical understanding, and asks what authorship means when models can generate proofs at scale.
The closing argument puts human responsibility upstream: choosing definitions, judging abstractions, curating Mathlib, inspecting certificates, and deciding which problems deserve effort. He also reflects on Lean's future beyond its founder and gives a practical starting point for new users.
---
TIMESTAMPS:
00:00:00 Cold open: the green checkmark can lie
00:00:59 Cathedral or bazaar: who controls Lean's core?
00:04:55 Why Lean's core stays small and protected
00:08:07 The Slack purge, Brandolini's law and the Lean FRO
00:11:12 The Collatz exploit: two kernels, two bugs
00:16:44 More kernels, reward hacking and safety by transparency
00:21:04 Sponsor: Parallel
00:21:59 Kim Morrison, Claude and the zlib proof
00:25:25 Can we specify complex systems?
00:28:04 Specs change: proofs are cheaper to redo with AI
00:31:27 From Lean 1 to Lean 4
00:34:57 Dependent types in plain terms
00:37:25 Lean 4's extensibility and Mathlib's growth
00:41:44 Mathlib as infrastructure: Formal Frontiers
00:44:15 Creativity, abstraction and nut-sniping
00:48:46 Breadcrumbs, not learning: what AI agents lack
00:53:03 Competence without comprehension, and verified guardrails
00:56:21 Is the human still the author?
01:01:44 AlphaProof, LLMs and why certificates still matter
01:06:38 What's next for Lean, and its legacy
01:11:42 How to start learning Lean
---
REFERENCES:
tool:
[00:00:48] Lean
lean-lang.org
[00:01:24] Mathlib
github.com/leanprover-community/mathlib4
[00:12:09] nanoda_lib
github.com/ammkrn/nanoda_lib
[00:12:19] CollatzLean
github.com/xrchz/CollatzLean/blob/a79357462a33d2a6babd4cf6c8d8bcd25425d653/README.md
[00:13:06] Lean issue 14576
github.com/leanprover/lean4/issues/14576
[00:13:21] Lean pull request 14577
github.com/leanprover/lean4/pull/14577
[00:13:45] nanoda_lib pull request 22
github.com/ammkrn/nanoda_lib/pull/22
[00:15:33] Lean comparator
github.com/leanprover/comparator
[00:18:02] Lean4Lean
github.com/digama0/lean4lean
[00:19:50] ARC-AGI-3
arcprize.org/arc-agi/3
[00:21:59] lean-zip
github.com/kim-em/lean-zip
[00:29:45] CompCert
compcert.org
[00:29:45] seL4
sel4.org
[00:30:23] Z3
github.com/Z3Prover/z3
[00:38:10] Veil
github.com/verse-lab/veil
[00:38:10] Velvet
github.com/verse-lab/velvet
organization:
[00:00:52] Lean FRO
lean-lang.org/fro
[00:42:39] Mathlib Initiative
mathlib-initiative.org/about
[01:11:42] Computerphile
youtube.com/@Computerphile
person:
[00:05:11] Ilya Sergey
ilyasergey.net
[00:10:31] Joachim Breitner
joachim-breitner.de
[00:32:58] Adam Chlipala
adam.chlipala.net
[00:38:42] Kevin Buzzard
ma.imperial.ac.uk/~buzzard
[01:10:16] Terence Tao
terrytao.wordpress.com
book:
[00:08:19] The Proof in the Code
us.macmillan.com/books/9780374620059/theproofinthecode
other:
[00:10:05] Brandolini's law
en.wikipedia.org/wiki/Brandolini%27s_law
[00:45:33] A new result on unit distances
openai.com/index/model-disproves-discrete-geometry-conjecture
[01:01:08] Fermat's Last Theorem formalisation
imperialcollegelondon.github.io/FLT
paper:
[00:34:36] The Lean 4 theorem prover and programming language
doi.org/10.1007/978-3-030-79876-5_37
[01:02:06] AlphaProof
doi.org/10.1038/s41586-025-09833-y
[01:04:24] AlphaZero
arxiv.org/abs/1712.01815
---
RESCRIPT:
app.rescript.info/share/7d3d4a0059443236a01f6c9acbf4db58
app.rescript.info/api/public/sessions/b007264c0ce89047/pdf
Leonardo de Moura created Lean and co-created Z3.
---
This episode is sponsored by Parallel.
Parallel, where agents find answers: web search, extraction and deep research APIs built for AI agents.
Start free with the Parallel MCP server and $5 of credits every month: parallel.ai/mlst?utm_source=creator&utm_medium=podcast&utm_content=MLST
---
Tim Scarfe talks with Leo about how Lean escaped its original audience, why dependent types and Mathlib made it useful to working mathematicians, and what happens when formal verification leaves the lab. De Moura explains the small trusted kernel and independent checkers, and gives his account of the recent Collatz incident, in which a purported proof was accepted by both Lean's official kernel and Nanoda, apparently by exploiting a different bug in each.
The conversation then turns to AI-produced proofs and software. Claude agents rebuild zlib in Lean and the result is verified, yet the example exposes the loophole at the centre of the show: a proof only certifies the specification humans chose to write. De Moura describes proof search as a game, separates formal competence from mathematical understanding, and asks what authorship means when models can generate proofs at scale.
The closing argument puts human responsibility upstream: choosing definitions, judging abstractions, curating Mathlib, inspecting certificates, and deciding which problems deserve effort. He also reflects on Lean's future beyond its founder and gives a practical starting point for new users.
---
TIMESTAMPS:
00:00:00 Cold open: the green checkmark can lie
00:00:59 Cathedral or bazaar: who controls Lean's core?
00:04:55 Why Lean's core stays small and protected
00:08:07 The Slack purge, Brandolini's law and the Lean FRO
00:11:12 The Collatz exploit: two kernels, two bugs
00:16:44 More kernels, reward hacking and safety by transparency
00:21:04 Sponsor: Parallel
00:21:59 Kim Morrison, Claude and the zlib proof
00:25:25 Can we specify complex systems?
00:28:04 Specs change: proofs are cheaper to redo with AI
00:31:27 From Lean 1 to Lean 4
00:34:57 Dependent types in plain terms
00:37:25 Lean 4's extensibility and Mathlib's growth
00:41:44 Mathlib as infrastructure: Formal Frontiers
00:44:15 Creativity, abstraction and nut-sniping
00:48:46 Breadcrumbs, not learning: what AI agents lack
00:53:03 Competence without comprehension, and verified guardrails
00:56:21 Is the human still the author?
01:01:44 AlphaProof, LLMs and why certificates still matter
01:06:38 What's next for Lean, and its legacy
01:11:42 How to start learning Lean
---
REFERENCES:
tool:
[00:00:48] Lean
lean-lang.org
[00:01:24] Mathlib
github.com/leanprover-community/mathlib4
[00:12:09] nanoda_lib
github.com/ammkrn/nanoda_lib
[00:12:19] CollatzLean
github.com/xrchz/CollatzLean/blob/a79357462a33d2a6babd4cf6c8d8bcd25425d653/README.md
[00:13:06] Lean issue 14576
github.com/leanprover/lean4/issues/14576
[00:13:21] Lean pull request 14577
github.com/leanprover/lean4/pull/14577
[00:13:45] nanoda_lib pull request 22
github.com/ammkrn/nanoda_lib/pull/22
[00:15:33] Lean comparator
github.com/leanprover/comparator
[00:18:02] Lean4Lean
github.com/digama0/lean4lean
[00:19:50] ARC-AGI-3
arcprize.org/arc-agi/3
[00:21:59] lean-zip
github.com/kim-em/lean-zip
[00:29:45] CompCert
compcert.org
[00:29:45] seL4
sel4.org
[00:30:23] Z3
github.com/Z3Prover/z3
[00:38:10] Veil
github.com/verse-lab/veil
[00:38:10] Velvet
github.com/verse-lab/velvet
organization:
[00:00:52] Lean FRO
lean-lang.org/fro
[00:42:39] Mathlib Initiative
mathlib-initiative.org/about
[01:11:42] Computerphile
youtube.com/@Computerphile
person:
[00:05:11] Ilya Sergey
ilyasergey.net
[00:10:31] Joachim Breitner
joachim-breitner.de
[00:32:58] Adam Chlipala
adam.chlipala.net
[00:38:42] Kevin Buzzard
ma.imperial.ac.uk/~buzzard
[01:10:16] Terence Tao
terrytao.wordpress.com
book:
[00:08:19] The Proof in the Code
us.macmillan.com/books/9780374620059/theproofinthecode
other:
[00:10:05] Brandolini's law
en.wikipedia.org/wiki/Brandolini%27s_law
[00:45:33] A new result on unit distances
openai.com/index/model-disproves-discrete-geometry-conjecture
[01:01:08] Fermat's Last Theorem formalisation
imperialcollegelondon.github.io/FLT
paper:
[00:34:36] The Lean 4 theorem prover and programming language
doi.org/10.1007/978-3-030-79876-5_37
[01:02:06] AlphaProof
doi.org/10.1038/s41586-025-09833-y
[01:04:24] AlphaZero
arxiv.org/abs/1712.01815
---
RESCRIPT:
app.rescript.info/share/7d3d4a0059443236a01f6c9acbf4db58
app.rescript.info/api/public/sessions/b007264c0ce89047/pdf


![Strange Geometric Shapes Found Inside AIs — Tom McGrath
Tom McGrath is co-founder and Chief Scientist at Goodfire, and a former Google DeepMind researcher. He joins Tim Scarfe to ask what neural networks actually learn, whether their internal representations converge on structures in the world, and whether interpretability can extract new scientific knowledge rather than merely explain model outputs.
Beginning with AlphaZero and learned modularity, the conversation moves into neural geometry: concept manifolds, reusable computation inside Llama, and why activation steering can fail when it pushes a model off-manifold. McGrath then makes the case for intentional design, using interpretability as part of the training loop. They examine controlled generalisation, features as rewards, predictive data debugging, and the uncomfortable fact that a model may recognise a hallucination or reward hack and still produce it.
The discussion closes on grader awareness, oversight and collusion between adaptive agents, then returns to sparse autoencoders. SAEs are useful, McGrath argues, but they may fracture the higher-dimensional structures networks actually use. This episode was made with support from Goodfire.
TIMESTAMPS:
00:00:00 Introduction: Can interpretability speed-run science?
00:02:03 The invisible grader
00:06:51 What AlphaZero learned from the world
00:12:24 Interpretability as a control loop
00:21:54 The forbidden method and safer interventions
00:37:36 Why models catch hallucinations too late
00:46:19 Debug the dataset before training
00:50:44 Why neural networks become modular
00:55:57 Finding the geometry inside a network
01:02:55 Why steering falls off the manifold
01:12:10 A reusable calculator inside Llama
01:17:19 From abstractions to goals
01:25:28 Reward hacking, oversight and collusion
01:37:23 Are sparse autoencoders dead?
REFERENCES:
paper:
[00:05:45] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
https://arxiv.org/abs/2502.17424v7
[00:11:05] Acquisition of Chess Knowledge in AlphaZero
https://arxiv.org/abs/2111.09259
[00:25:30] Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning
https://arxiv.org/abs/2507.16795
[00:29:30] Persona Vectors: Monitoring and Controlling Character Traits in Language Models
https://arxiv.org/abs/2507.21509
[00:41:14] Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability
https://arxiv.org/abs/2602.10067
[00:47:03] Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal
https://arxiv.org/abs/2606.12360
[01:00:26] Do Sparse Autoencoders Capture Concept Manifolds?
https://arxiv.org/abs/2604.28119
[01:03:04] Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
https://arxiv.org/abs/2605.05115
[01:14:20] Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
https://arxiv.org/abs/2605.01148
[01:29:35] Measuring Reward-Seeking via Contrastive Belief Updates
https://arxiv.org/abs/2607.18966v1
other:
[00:15:44] Intentional Design
https://www.goodfire.com/blog/intentional-design
[00:56:12] The World Inside Neural Networks
https://www.goodfire.com/research/the-world-inside-neural-networks
[01:37:28] A Pragmatic Vision for Interpretability
https://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability
RESCRIPT:
https://app.rescript.info/share/846cfee4131b664fd09209cc3b98018e Strange Geometric Shapes Found Inside AIs — Tom McGrath](https://i.ytimg.com/vi/_egu7OFem-k/mqdefault.jpg)
![AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]
Dr. Ilia Shumailov - Former DeepMind AI Security Researcher, now building security tools for AI agents
Ever wondered what happens when AI agents start talking to each other—or worse, when they start breaking things? Ilia Shumailov spent years at DeepMind thinking about exactly these problems, and hes here to explain why securing AI is way harder than you think.
**SPONSOR MESSAGES**
—
Check out notebooklm for your research project, its really powerful
https://notebooklm.google.com/
—
Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
Were racing toward a world where AI agents will handle our emails, manage our finances, and interact with sensitive data 24/7. But there is a problem. These agents are nothing like human employees. They never sleep, they can touch every endpoint in your system simultaneously, and they can generate sophisticated hacking tools in seconds. Traditional security measures designed for humans simply wont work.
Dr. Ilia Shumailov
https://x.com/iliaishacked
https://iliaishacked.github.io/
https://sequrity.ai/
TRANSCRIPT:
https://app.rescript.info/public/share/dVGsk8dz9_V0J7xMlwguByBq1HXRD6i4uC5z5r7EVGM
More from Ilia on our Patreon:
https://www.patreon.com/posts/116142401/ (interview from last year)
https://www.patreon.com/posts/ilia-shumailov-140359158 (extended version of this interview)
TOC:
00:00:00 - Introduction & Trusted Third Parties via ML
00:03:45 - Background & Career Journey
00:06:42 - Safety vs Security Distinction
00:09:45 - Prompt Injection & Model Capability
00:13:00 - Agents as Worst-Case Adversaries
00:15:45 - Personal AI & CAML System Defense
00:19:30 - Agents vs Humans: Threat Modeling
00:22:30 - Calculator Analogy & Agent Behavior
00:25:00 - IMO Math Solutions & Agent Thinking
00:28:15 - Diffusion of Responsibility & Insider Threats
00:31:00 - Open Source Security Concerns
00:34:45 - Supply Chain Attacks & Trust Issues
00:39:45 - Architectural Backdoors
00:44:00 - Academic Incentives & Defense Work
00:48:30 - Semantic Censorship & Halting Problem
00:52:00 - Model Collapse: Theory & Criticism
00:59:30 - Career Advice & Ross Anderson Tribute
REFS:
Lessons from Defending Gemini Against Indirect Prompt Injections
https://arxiv.org/abs/2505.14534
Defeating Prompt Injections by Design. Google, Google DeepMind, and ETH Zurich. (CAML)
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F.
https://arxiv.org/pdf/2503.18813
Agentic Misalignment: How LLMs could be insider threats
https://www.anthropic.com/research/agentic-misalignment
STOP ANTHROPOMORPHIZING INTERMEDIATE TOKENS AS REASONING/THINKING TRACES!
Subbarao Kambhampati et al
https://arxiv.org/pdf/2504.09762
Meiklejohn, S., Blauzvern, H., Maruseac, M., Schrock, S., Simon, L., & Shumailov, I. (2025).
Machine learning models have a supply chain problem.
https://arxiv.org/abs/2505.22778
Gao, Y., Shumailov, I., & Fawaz, K. (2025).
Supply-chain attacks in machine learning frameworks.
In Proceedings of the 8th MLSys Conference.
https://openreview.net/pdf?id=EH5PZW6aCr
Apache Log4j Vulnerability Guidance
https://www.cisa.gov/news-events/news/apache-log4j-vulnerability-guidance
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2023).
Architectural backdoors in neural networks.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 21163–21173).
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2022).
Architectural backdoors in neural networks. arXiv preprint arXiv:2206.07840.
https://arxiv.org/pdf/2206.07840
Langford, H., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2024).
Architectural neural backdoors from first principles.
arXiv preprint arXiv:2402.06957.
Küchler, N., Petrov, I., Grobler, C., & Shumailov, I. (2025).
Architectural backdoors for within-batch data stealing and model inference manipulation.
arXiv preprint arXiv:2505.18323.
Position: Fundamental Limitations of LLM Censorship Necessitate New Approaches
David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Papernot, Vardan Papyan
https://proceedings.mlr.press/v235/glukhov24a.html
AlphaEvolve MLST interview [Matej Balog, Alexander Novikov]
https://www.youtube.com/watch?v=vC9nAosXrJw AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]](https://i.ytimg.com/vi/aoX_pGQMbEM/mqdefault.jpg)

![The Weird ChatGPT Hack That Leaked Training Data [Dr. Yannic Kilcher / Prof. Florian Tramer]
Dr. Yannic lightspeed Kilcher sits down with an AI Security Researcher Prof. Florian Tramer (an expert in the field, collaborating with folks like Nicholas Carlini and Milad Nasr at Google) to ruminate on the murky state of affairs in AI security.
Out of the gate they assert that AI detectors (those tools that try to spot if text is human-written or AI-generated) are basically doomed to fail—like trying to catch an eel with greasy hands blindfolded. Lightspeed Kilcher shares a funny story about how the word Delve kept rearing its ugly head in AI text because of training data emanating from Nigerian crowd workers, leading to false flags in job applications.
Tramer then takes the gloves off about security risks, saying that even a 99% success rate in AI means total failure in security terms—attackers will always find that 1% weak spot. They share an interesting example of a weird attack on ChatGPT: asking it to repeat poem forever made it spit out memorized internet scraps, leaking training data like a sieve. OpenAI patched it, but its like slapping a band-aid on a shark bite.
Lightspeed Kilcher asks about top worries, and Tramer lists three big ones in relatable terms: 1) Training AI on private data (like medical records) could lead to massive leaks, since models memorize stuff without us fully understanding how; 2) Prompt injections, where bad actors hijack AI agents to do harmful things, like Anthropics demo that controls computers—its like giving your keys to a stranger; and 3) The rush to deploy AI without safeguards, echoing past tech blunders like SQL hacks.
They reflect on how ChatGPT has lit a fire under AI security research—making it amazing and scary by turning hypotheticals into real-world issues with millions of users. The conversation touches on why just scaling up AI wont fix everything (we need causal understanding, not just more data), and why watermarking AI outputs isnt a silver bullet—its easy to bypass, especially with open-source models.
Its about time we had our old comrade light speed Kilcher back on MLST!
TRANSCRIPT:
https://app.rescript.info/public/share/PGieXFZCRyJjJswicatmNFia_pKCt_WVP-pZS4qguiw
LINKS:
Scalable Extraction of Training Data from (Production) Language Models [Poem Attack]
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel
https://arxiv.org/abs/2311.17035
- Prof. Florian Tramer
https://www.floriantramer.com/
- Yannic Kilchers YouTube channel
https://www.youtube.com/c/YannicKilcher
- Nicholas Carlinis research profile (collaborator on the data recovery project):
https://nicholas.carlini.com/ – Includes papers on AI attacks and memorization risks. The Weird ChatGPT Hack That Leaked Training Data [Dr. Yannic Kilcher / Prof. Florian Tramer]](https://i.ytimg.com/vi/c_hmxRVDXBE/mqdefault.jpg)
![AI training data will never be fully synthetic [SPONSORED]
We sat down with Sara Saab (VP of Product at Prolific) and Enzo Blindow (VP of Data and AI at Prolific) to explore the critical role of human evaluation in AI development and the challenges of aligning AI systems with human values. Prolific is a human annotation and orchestration platform for AI used by many of the major AI labs. This is a sponsored show in partnership with Prolific.
**SPONSOR MESSAGES**
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
While technologists want to remove humans from the loop for speed and efficiency, these non-deterministic AI systems actually require more human oversight than ever before. Prolifics approach is to put well-treated, verified, diversely demographic humans behind an API - making human feedback as accessible as any other infrastructure service.
When AI models like Grok 4 achieve top scores on technical benchmarks but feel awkward or problematic to use in practice, it exposes the limitations of our current evaluation methods. The guests argue that optimizing for benchmarks may actually weaken model performance in other crucial areas, like cultural sensitivity or natural conversation.
We also discuss Anthropics research showing that frontier AI models, when given goals and access to information, independently arrived at solutions involving blackmail - without any prompting toward unethical behavior. Even more concerning, the more sophisticated the model, the more susceptible it was to this agentic misalignment.
Enzo and Sarah present Prolifics Humane leaderboard as an alternative to existing benchmarking systems. By stratifying evaluations across diverse demographic groups, they reveal that different populations have vastly different experiences with the same AI models.
Looking forwards the guests imagine a world where humans take on coaching and teaching roles for AI systems - similar to how we might correct a child or review code. Working conditions and the evolution of labor in an AI-augmented world is also important to consider. Rather than replacing humans entirely, we may be moving toward more sophisticated forms of human-AI collaboration.
We need more representative evaluation frameworks that capture the messy reality of human values and cultural diversity.
Visit Prolific:
https://www.prolific.com/
Sara Saab (VP Product):
https://uk.linkedin.com/in/sarasaab
Enzo Blindow (VP Data & AI):
https://uk.linkedin.com/in/enzoblindow
TRANSCRIPT:
https://app.rescript.info/public/share/xZ31-0kJJ_xp4zFSC-bunC8-hJNkHpbm7Lg88RFcuLE
TOC:
[00:00:00] Intro & Background
[00:03:16] Human-in-the-Loop Challenges
[00:17:19] Can AIs Understand?
[00:32:02] Benchmarking & Vibes
[00:51:00] Agentic Misalignment Study
[01:03:00] Data Quality vs Quantity
[01:16:00] Future of AI Oversight
REFS:
Anthropic Agentic Misalignment
https://www.anthropic.com/research/agentic-misalignment
Value Compass
https://arxiv.org/pdf/2409.09586
Reasoning Models Don’t Always Say What They Think (Anthropic)
https://www.anthropic.com/research/reasoning-models-dont-say-think
https://assets.anthropic.com/m/71876fabef0f0ed4/original/reasoning_models_paper.pdf
Maslow’s Hierarchy Of Needs
https://www.simplypsychology.org/maslow.html
Apollo research - science of evals blog post
https://www.apolloresearch.ai/blog/we-need-a-science-of-evals
Leaderboard Illusion
https://www.youtube.com/watch?v=9W_OhS38rIE MLST video
The Leaderboard Illusion [2025]
Shivalika Singh, Yiyang Nan, Alex Wang, Daniel DSouza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, Sara Hooker
https://arxiv.org/abs/2504.20879
Humanities last exam
https://arxiv.org/abs/2501.14249
PRISM paper
https://arxiv.org/abs/2405.10254
https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback
Collective intelligence project
https://www.cip.org/whitepaper
https://ischool.utoronto.ca/faculty-profile/brian-cantwell-smith/
Ghost work (Mary Gray)
https://www.amazon.com/Ghost-Work-Silicon-Building-Underclass/dp/1328566242
Fairwork Cloudwork report
https://fair.work/en/ratings/cloudwork/
Gibson theory of affordances
https://cs.brown.edu/courses/cs137/2017/readings/Gibson-AFF.pdf AI training data will never be fully synthetic [SPONSORED]](https://i.ytimg.com/vi/cnxZZTl1tkk/mqdefault.jpg)

![A Physicist Found the Hidden Phase Transitions in Society — Cristopher Moore
We are joined by Cristopher Moore, a professor at the Santa Fe Institute with a diverse background in physics, computer science, and machine learning.
The conversation begins with Cristopher, who calls himself a frog explaining that he prefers to dive deep into specific, concrete problems rather than taking a high-level birds-eye view.
They explore why current AI models, like transformers, are so surprisingly effective. Cristopher argues its because the real world isnt random; its full of rich structures, patterns, and hierarchies that these models can learn to exploit, even if we dont fully understand how.
ADDITIONAL FREE CONTENT:
We also filmed Cris talk at DISI, the audio quality was too bad for us to officially publish, slides included
https://www.dropbox.com/scl/fo/bn0tn7z9fxrh1rya1ycrt/AH_d4SOOvlauxLzBgfnNUA0?rlkey=ec65ero4feo5uezum0j6p9z4t&dl=0
**SPONSORS**
Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy (note we got url wrong in the video, this is correct URL!)
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
***
Cristopher Moore:
https://sites.santafe.edu/~moore/
TOC:
00:00:00 - Introduction
00:02:05 - Meet Christopher Moore: A Frog in the World of Science
00:05:14 - The Limits of Transformers and Real-World Data
00:11:19 - Intelligence as Creative Problem-Solving
00:23:30 - Grounding, Meaning, and Shared Reality
00:31:09 - The Nature of Creativity and Aesthetics
00:44:31 - Computational Irreducibility and Universality
00:53:06 - Turing Completeness, Recursion, and Intelligence
01:11:26 - The Universe Through a Computational Lens
01:26:45 - Algorithmic Justice and the Need for Transparency
TRANSCRIPT: https://app.rescript.info/public/share/VRe2uQSvKZOm0oIBoDsrNwt46OMCqRnShVnUF3qyoFk
Filmed at DISI (Diverse Intelligences Summer Institute)
https://disi.org/
REFS:
The Nature of computation [Chris Moore]
https://nature-of-computation.org/
Birds and Frogs [Freeman Dyson]
https://www.ams.org/notices/200902/rtx090200212p.pdf
Replica Theory [Parisi et al]
https://arxiv.org/pdf/1409.2722
Janossy pooling [Fabian Fuchs]
https://fabianfuchsml.github.io/equilibriumaggregation/
Cracking the cryptic [YT channel]
https://www.youtube.com/c/CrackingTheCryptic
Sudoko Bench [Sakana]
https://sakana.ai/sudoku-bench/
Fractured entangled representations “phylogenetic locking in comment” [Kumar/Stanley]
https://arxiv.org/pdf/2505.11581 (see our shows on this)
The War Against Cliché: [Martin Amis]
https://www.amazon.com/War-Against-Cliche-Reviews-1971-2000/dp/0375727167
Rule 110 (CA)
https://mathworld.wolfram.com/Rule150.html
Universality in Elementary Cellular Automata [Matt Cooke]
https://wpmedia.wolfram.com/sites/13/2018/02/15-1-1.pdf
Small Semi-Weakly Universal Turing Machines [Damien Woods]
https://tilde.ini.uzh.ch/users/tneary/public_html/WoodsNeary-FI09.pdf
COMPUTING MACHINERY AND INTELLIGENCE [Turing, 1950]
https://courses.cs.umbc.edu/471/papers/turing.pdf
Comment on Space Time as a causal set [Moore, 88]
https://sites.santafe.edu/~moore/comment.pdf
Recursion Theory on the Reals and Continuous-time Computation [Moore, 96] A Physicist Found the Hidden Phase Transitions in Society — Cristopher Moore](https://i.ytimg.com/vi/dB9lJkUkIUM/mqdefault.jpg)
![The Dangerous Illusion of AI Coding? - Jeremy Howard
Dive into the realities of AI-assisted coding, the origins of modern fine-tuning, and the cognitive science behind machine learning with fast.ai founder Jeremy Howard. In this episode, we unpack why AI might be turning software engineering into a slot machine and how to maintain true technical intuition in the age of large language models.
GTC is coming, the premier AI conference, great opportunity to learn about AI. NVIDIA and partners will showcase breakthroughs in physical AI, AI factories, agentic AI, and inference, exploring the next wave of AI innovation for developers and researchers. Register for virtual GTC for free, using my link and win NVIDIA DGX Spark (https://nvda.ws/4qQ0LMg)
Jeremy Howard is a renowned data scientist, researcher, entrepreneur, and educator. As the co-founder of fast.ai, former President of Kaggle, and the creator of ULMFiT, Jeremy has spent decades democratizing deep learning. His pioneering work laid the foundation for modern transfer learning and the pre-training and fine-tuning paradigm that powers todays language models.
Key Topics and Main Insights Discussed:
- The Origins of ULMFiT and Fine-Tuning
- The Vibe Coding Illusion and Software Engineering
- Cognitive Science, Friction, and Learning
- The Future of Developers
RESCRIPT: https://app.rescript.info/public/share/BhX5zP3b0m63srLOQDKBTFTooSzEMh_ARwmDG_h_izk
https://app.rescript.info/api/public/sessions/62d06c0336c567d6/pdf
Jeremy Howard:
https://x.com/jeremyphoward
https://www.answer.ai/
TIMESTAMPS (fixed):
00:00:00 Introduction & GTC Sponsor
00:04:30 ULMFiT & The Birth of Fine-Tuning
00:12:00 Intuition & The Mechanics of Learning
00:18:30 Abstraction Hierarchies & AI Creativity
00:23:00 Claude Code & The Interpolation Illusion
00:27:30 Coding vs. Software Engineering
00:30:00 Cosplaying Intelligence: Dennett vs. Searle
00:36:30 Automation, Radiology & Desirable Difficulty
00:42:30 Organizational Knowledge & The Slope
00:48:00 Vibe Coding as a Slot Machine
00:54:00 The Erosion of Control in Software
01:01:00 Interactive Programming & REPL Environments
01:05:00 The Notebook Debate & Exploratory Science
01:17:30 AI Existential Risk & Power Centralization
01:24:20 Current Risks, Privacy & Enfeeblement
REFERENCES:
Blog Post:
[00:03:00] fast.ai Blog: Self-Supervised Learning
https://www.fast.ai/posts/2020-01-13-self_supervised.html
[00:13:30] DeepMind Blog: Gemini Deep Think
https://deepmind.google/blog/accelerating-mathematical-and-scientific-discovery-with-gemini-deep-think/
[00:19:30] Modular Blog: Claude C Compiler analysis
https://www.modular.com/blog/the-claude-c-compiler-what-it-reveals-about-the-future-of-software
[00:19:45] Anthropic Engineering Blog: Building C Compiler
https://www.anthropic.com/engineering/building-c-compiler
[00:48:00] Cursor Blog: Scaling Agents
https://cursor.com/blog/scaling-agents
[01:05:15] fast.ai Blog: NB Dev Merged Driver
https://www.fast.ai/posts/2022-08-25-jupyter-git.html
[01:17:30] Jeremy Howard: Response to AI Risk Letter
https://www.normaltech.ai/p/is-avoiding-extinction-from-ai-really
Book:
[00:08:30] M. Chirimuuta: The Brain Abstracted
https://mitpress.mit.edu/9780262548045/the-brain-abstracted/
[00:30:00] Daniel Dennett: Consciousness Explained
https://www.amazon.com/Consciousness-Explained-Daniel-C-Dennett/dp/0316180661
[00:42:30] Cesar Hidalgo: Infinite Alphabet / Laws of Knowledge
https://www.amazon.com/Infinite-Alphabet-Laws-Knowledge/dp/0241655676
Archive Article:
[00:13:45] MLST Archive: Why Creativity Cannot Be Interpolated
https://archive.mlst.ai/read/why-creativity-cannot-be-interpolated
Research Study:
[00:24:30] METR Study: AI OS Development
https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
Paper:
[00:24:45] Fred Brooks: No Silver Bullet
https://www.cs.unc.edu/techreports/86-020.pdf
[00:30:15] John Searle: Minds, Brains, and Programs
https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/minds-brains-and-programs/DC644B47A4299C637C89772FACC2706A
Research Paper:
[00:13:50] Mathilde Caron et al.: Emerging Properties in Self-Supervised Vision Transformers (DINO)
https://arxiv.org/abs/2104.14294
[00:25:00] Oxford VGG: Sculptor Identification Paper
https://www.robots.ox.ac.uk/~vgg/publications/2012/Arandjelovic12a/arandjelovic12a.pdf
[00:36:30] Anthropic Paper: AI Skill Formation
https://arxiv.org/pdf/2601.20245
Historical Reference:
[00:36:45] Ebbinghaus: Memory / Spaced Repetition
https://www.loc.gov/item/e11000616/
Technical Note:
[00:42:45] John Ousterhout: Slope vs Intercept
https://gist.github.com/gtallen1187/e83ed02eac6cc8d7e185
Video:
[00:59:00] Bret Victor: Inventing on Principle
https://vimeo.com/906418692
[01:05:00] Joel Grus: I Dont Like Notebooks
https://www.youtube.com/watch?v=7jiPeIFXb6U The Dangerous Illusion of AI Coding? - Jeremy Howard](https://i.ytimg.com/vi/dHBEQ-Ryo24/mqdefault.jpg)
![We Built Calculators Because Were STUPID! [Prof. David Krakauer]
Prof. David Krakauer argues that intelligence isnt about knowing more—its about doing more with less.
His core thesis flips our assumptions on their head: while emergence in complex systems is about more is different, intelligence is fundamentally about less is more.
Why Large Language Models Miss the Point
LLMs are more is more, Krakauer explains. Theyre essentially sophisticated libraries that know everything but understand nothing. When a student looks up answers in a library during an exam, we call them a cheater, not intelligent.
Intelligence Across All Life
Perhaps most surprisingly, Krakauer claims that all living things are intelligent—even bacteria. His reasoning: intelligence is the information accumulated through evolutions selective pressures. Its why we can say an elephant is smarter than a worm, but wed never say its more alive. The difference is accumulated capacity.
Three Dimensions of Intelligence
Krakauer proposes intelligence exists in three distinct flavours:
- Strategic Intelligence - Adaptation and survival where viruses actually beat us/
- Inferential Intelligence - Math and computation (where we built calculators because were so bad at it)
- Representational Intelligence - Finding better ways to encode problems (the most uniquely human)
The Soma Cube Insight
Using a Soma Cube puzzle, Krakauer demonstrates something incredible: a four-year-old can solve a combinatorial problem with 15,000 logical clauses—not by computing, but by using physical representation. The physical world does the computing for us. This is intelligence: making impossibly hard problems tractable through clever representation.
This talk was recorded at DISI 2025 https://disi.org/
David Krakauer
President and William H. Miller Professor of Complex Systems
https://www.santafe.edu/people/profile/david-krakauer
Note this version is edited down, you can watch the full version on our Patreon - https://www.patreon.com/posts/masterclass-on-142898847
Watch interview we published with David - https://www.youtube.com/watch?v=jXa8dHzgV8U
TRANSCRIPT:
https://app.rescript.info/public/share/Cs8QFwMw_FiOEROkNPiIf6Y-e_ahAxNMl0oFiL7_4QA
TOC:
00:00:00 - Complexity, Life & Intelligence
00:00:45 - Ptolemy vs Newton
00:01:30 - Purpose of Science
00:02:15 - Feynman on Rules vs Strategies
00:03:00 - Entropy as Unifying Rule
00:03:45 - Intelligence in Physics
00:04:15 - Complex Systems & Broken Symmetries
00:05:00 - Historical Definitions of Intelligence
00:06:15 - Herbert: Capacity to Acquire Capacity
00:06:45 - Problems with Turing Test
00:07:30 - Intelligence vs Knowledge
00:08:00 - LLMs as Libraries
00:08:30 - Humanistic Perspectives
00:09:00 - Unified Theory: Life & Intelligence
00:09:30 - Intensive vs Extensive Properties
00:10:30 - Selection Gradient Analogy
00:11:15 - Universal Intelligence (Bacteria to Elephants)
00:12:30 - Three Dimensions of Intelligence
00:13:00 - Strategic Intelligence & Viruses
00:13:30 - Inferential Intelligence & Calculators
00:14:15 - Encoding & Representation
00:14:30 - Different Paths: Life vs AI
00:15:15 - Soma Cube Problem
00:16:00 - Principle of Materiality
00:17:15 - Embodied Representation
00:17:45 - Intelligence Makes Hard Problems Easy
00:18:00 - Stupidity & Conclusion
REFS:
Papers and Academic Works
More is Different by Phil Anderson [https://www.tkm.kit.edu/downloads/TKM1_2011_more_is_different_PWA.pdf ]
Recent paper with Melanie and John Krakauer on emergence [https://arxiv.org/abs/2506.11135 ]
Formal treatment paper on intelligence/evolution duality from David - Darwinian demons, evolutionary complexity, and information maximization [https://wiki.santafe.edu/images/b/b1/CHAOEH213037110_1.pdf ]
The Unreasonable Effectiveness of Mathematics in the Natural Sciences by Eugene Wigner [https://webhomes.maths.ed.ac.uk/~v1ranick/papers/wigner.pdf ] We Built Calculators Because Were STUPID! [Prof. David Krakauer]](https://i.ytimg.com/vi/dY46YsGWMIc/mqdefault.jpg)