Updated September 2026, 2 days ago
Machine Learning Street Talk
|
#68 DR. WALID SABA 2.0 - Natural Language Understanding [UNPLUGGED]
🔇 TAP TO UNMUTE
![#68 DR. WALID SABA 2.0 - Natural Language Understanding [UNPLUGGED]](https://i.ytimg.com/vi/pMtk-iUaEuQ/mqdefault.jpg "#68 DR. WALID SABA 2.0 - Natural Language Understanding [UNPLUGGED]
#68 DR. WALID SABA 2.0 - Natural Language Understanding [UNPLUGGED]")
![Why Every Brain Metaphor in History Has Been Wrong [SPECIAL EDITION]](https://i.ytimg.com/vi/pO0WZsN8Oiw/mqdefault.jpg "Why Every Brain Metaphor in History Has Been Wrong [SPECIAL EDITION]
What if everything we think we know about the brain is just a really good metaphor that we forgot was a metaphor?
This episode takes you on a journey through the history of scientific simplification, from a young Karl Friston watching wood lice in his garden to the bold claims that your mind is literally software running on biological hardware.
We bring together some of the most brilliant minds weve interviewed — Professor Mazviita Chirimuuta, Francois Chollet, Joscha Bach, Professor Luciano Floridi, Professor Noam Chomsky, Nobel laureate John Jumper, and more — to wrestle with a deceptively simple question: *When scientists simplify reality to study it, what gets captured and what gets lost?*
*Key ideas explored:*
*The Spherical Cow Problem* — Science requires simplification. Were limited creatures trying to understand systems far more complex than our working memory can hold. But when does a useful model become a dangerous illusion?
*The Kaleidoscope Hypothesis* — Francois Chollets beautiful idea that beneath all the apparent chaos of reality lies simple, repeating patterns — like bits of colored glass in a kaleidoscope creating infinite complexity. Is this profound truth or Platonic wishful thinking?
*Is Software Really Spirit?* — Joscha Bach makes the provocative claim that software is literally spirit, not metaphorically. We push back hard on this, asking whether the sameness we see across different computers running the same program exists in nature or only in our descriptions.
*The Cultural Illusion of AGI* — Why does artificial general intelligence seem so inevitable to people in Silicon Valley? Professor Chirimuuta suggests we might be caught in a cultural historical illusion — our mechanistic assumptions about minds making AI seem like destiny when it might just be a bet.
*Prediction vs. Understanding* — Nobel Prize winner John Jumper: AI can predict and control, but understanding requires a human in the loop.
Throughout history, weve described the brain as hydraulic pumps, telegraph networks, telephone switchboards, and now computers. Each metaphor felt obviously true at the time. This episode asks: what will we think was naive about our current assumptions in fifty years?
Featuring insights from *The Brain Abstracted* by Mazviita Chirimuuta — possibly the most influential book on how we think about thinking in 2025.
TIMESTAMPS:
00:00:00 The Wood Louse & The Spherical Cow
00:02:04 The Necessity of Abstraction
00:04:42 Simplicius vs. Ignorantio: The Boxing Match
00:06:39 The Kaleidoscope Hypothesis
00:08:40 Is the Mind Software?
00:13:15 Critique of Causal Patterns
00:14:40 Temperature is Not a Thing
00:18:24 The Ship of Theseus & Ontology
00:23:45 Metaphors Hardening into Reality
00:25:41 The Illusion of AGI Inevitability
00:27:45 Prediction vs. Understanding
00:32:00 Climbing the Mountain vs. The Helicopter
00:34:53 Haptic Realism & The Limits of Knowledge
REFERENCES:
Person:
[00:00:00] Karl Friston (UCL)
https://profiles.ucl.ac.uk/1236-karl-friston
[00:06:30] Francois Chollet
https://fchollet.com/
[00:14:41] Cesar Hidalgo, MLST interview.
https://www.youtube.com/watch?v=vzpFOJRteeI
[00:30:30] Terence Taos Blog
https://terrytao.wordpress.com/
Book:
[00:02:25] The Brain Abstracted
https://mitpress.mit.edu/9780262548045/the-brain-abstracted/
[00:06:00] On Learned Ignorance
https://www.amazon.com/Nicholas-Cusa-learned-ignorance-translation/dp/0938060236
[00:24:15] Science and the Modern World
https://amazon.com/dp/0684836394
Interview.:
[00:02:43] The Brain Abstracted Patreon interview
https://www.patreon.com/posts/brain-abstracted-124479979
Interview:
[00:04:18] David Krakauers presentation on intelligence.
https://www.youtube.com/watch?v=dY46YsGWMIc
[00:06:45] Machine Learning Street Talk interview with Francois Chollet.
https://www.youtube.com/watch?v=JTU8Ha4Jyfc
[00:09:32] Joscha Bach Patreon interview.
https://www.patreon.com/posts/joscha-bach-deep-141884561
[00:18:24] Luciano Floridi, MLST interview.
https://www.youtube.com/watch?v=YLNGvvgq3eg
[00:25:02] Jeff Beck, MLST interview.
https://www.youtube.com/watch?v=9suqiofCiwM
[00:28:00] John Jumper
https://www.patreon.com/posts/john-jumper-on-144557652
[00:30:48] Noam Chomsky, MLST interview.
https://www.youtube.com/watch?v=axuGfh4UR9Q
[00:31:59] Anna Ciaunica
https://www.patreon.com/posts/dr-anna-ciaunica-144509970
[00:33:12] Mike Israetel debate on functionalism.
https://www.youtube.com/watch?v=4yYcN_mFi18
Company/Org:
[00:09:25] Neuralink
https://neuralink.com/
Paper:
[00:23:45] A Logical Calculus of Ideas Immanent in Nervous Activity
https://link.springer.com/article/10.1007/BF02478259
[00:28:00] Highly Accurate Protein Structure Prediction with AlphaFold
https://www.nature.com/articles/s41586-021-03819-2
Thank you to Dr. Maxwell Ramstead for early script work on this show (Ph.D student of Friston) and the woodlice story came from him!")
![Google Researcher Shows Life Emerges From Code [Blaise Agüera y Arcas]](https://i.ytimg.com/vi/rMSEqJ_4EBk/mqdefault.jpg "Google Researcher Shows Life Emerges From Code [Blaise Agüera y Arcas]
Blaise Agüera y Arcas explores some mind-bending ideas about what intelligence and life really are—and why they might be more similar than we think (filmed at ALIFE conference, 2025 - https://2025.alife.org/ ).
Life and intelligence are both fundamentally computational (he says). From the very beginning, living things have been running programs. Your DNA? Its literally a computer program, and the ribosomes in your cells are tiny universal computers building you according to those instructions.
**SPONSOR MESSAGES**
—
Prolific - Quality data. From real people. For faster breakthroughs.
https://www.prolific.com/?utm_source=mlst
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
Blaise argues that there is more to evolution than random mutations (like most people think). The secret to increasing complexity is *merging* i.e. when different organisms or systems come together and combine their histories and capabilities.
Blaise describes his BFF experiment where random computer code spontaneously evolved into self-replicating programs, showing how purpose and complexity can emerge from pure randomness through computational processes.
https://en.wikipedia.org/wiki/Blaise_Ag%C3%BCera_y_Arcas
https://x.com/blaiseaguera?lang=en
TRANSCRIPT:
https://app.rescript.info/public/share/VX7Gktfr3_wIn4Bj7cl9StPBO1MN4R5lcJ11NE99hLg
TOC:
00:00:00 Introduction - New book What is Intelligence?
00:01:45 Life as computation - Von Neumanns insights
00:12:00 BFF experiment - How purpose emerges
00:26:00 Symbiogenesis and evolutionary complexity
00:40:00 Functionalism and consciousness
00:49:45 AI as part of collective human intelligence
00:57:00 Comparing AI and human cognition
REFS:
What is intelligence [Blaise Agüera y Arcas]
https://whatisintelligence.antikythera.org/ [Read free online, interactive rich media]
https://mitpress.mit.edu/9780262049955/what-is-intelligence/ [MIT Press]
Large Language Models and Emergence: A Complex Systems Perspective
https://arxiv.org/abs/2506.11135
Our first Noam Chomsky MLST interview
https://www.youtube.com/watch?v=axuGfh4UR9Q
Chance and Necessity [Jacques Monod]
https://monoskop.org/images/9/99/Monod_Jacques_Chance_and_Necessity.pdf
Wonderful Life: The Burgess Shale and the History of Nature [Stephen Jay Gould]
https://www.amazon.co.uk/Wonderful-Life-Burgess-Nature-History/dp/0099273454
The major evolutionary transitions [E Szathmáry, J M Smith]
https://wiki.santafe.edu/images/0/0e/Szathmary.MaynardSmith_1995_Nature.pdf
Dont Sleep, There Are Snakes: Life and Language in the Amazonian Jungle [Dan Everett]
https://www.amazon.com/Dont-Sleep-There-Are-Snakes/dp/0307386120
The Nature of Technology: What It Is and How It Evolves [W. Brian Arthur]
https://www.amazon.com/Nature-Technology-What-How-Evolves-ebook/dp/B002RI9W16/
The MANIAC [Benjamin Labatut]
https://www.amazon.com/MANIAC-Benjam%C3%ADn-Labatut/dp/1782279814
When We Cease to Understand the World [Benjamin Labatut]
https://www.amazon.com/When-We-Cease-Understand-World/dp/1681375664/
The Boys in the Boat [Dan Brown]
https://www.amazon.com/Boys-Boat-Americans-Berlin-Olympics/dp/0143125478
How something can be said about Telling More Than We Can Know [Petter Johansson] (Split brain)
https://www.lucs.lu.se/fileadmin/user_upload/lucs/2011/01/Johansson-et-al.-2006-How-Something-Can-Be-Said-About-Telling-More-Than-We-Can-Know.pdf
If Anyone Builds It, Everyone Dies [Eliezer Yudkowsky, Nate Soares]
https://www.amazon.com/Anyone-Builds-Everyone-Dies-Superhuman/dp/0316595640
The science of cycology: Failures to understand how everyday objects work [Rebeca Lawson]
https://link.springer.com/content/pdf/10.3758/bf03195929.pdf
SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction [Kushin Mukherjee, Judith Fan et al]
https://arxiv.org/pdf/2312.03035
Brain-Score [Martin Schrimpf]
https://www.biorxiv.org/content/10.1101/407007v1.full.pdf
Feature Visualization [Chris Olah]
https://distill.pub/2017/feature-visualization/
THE REVERSAL CURSE [Lukas Berglund]
https://arxiv.org/pdf/2309.12288")
![Why High Benchmark Scores Don’t Mean Better AI [SPONSORED]](https://i.ytimg.com/vi/rqiC9a2z8Io/mqdefault.jpg "Why High Benchmark Scores Don’t Mean Better AI [SPONSORED]
Is a car that wins a Formula 1 race the best choice for your morning commute? Probably not. In this sponsored deep dive with Prolific, we explore why the same logic applies to Artificial Intelligence. While models are currently shattering records on technical exams, they often fail the most important test of all: *the human experience.*
Why High Benchmark Scores Don’t Mean Better AI
Joining us are *Andrew Gordon* (Staff Researcher in Behavioral Science) and *Nora Petrova* (AI Researcher) from *Prolific* . They reveal the hidden flaws in how we currently rank AI and introduce a more rigorous, humane way to measure whether these models are actually helpful, safe, and relatable for real people.
Key Insights in This Episode:
* *The F1 Car Analogy:* Andrew explains why a model that excels at the Humanities Last Exam might be a nightmare for daily use. Technical benchmarks often ignore the nuances of human communication and adaptability.
* *The Wild West of AI Safety:* As users turn to AI for sensitive topics like mental health, Nora highlights the alarming lack of oversight and the thin veneer of safety training—citing recent controversial incidents like Grok-3’s Mecha Hitler.
* *Fixing the Leaderboard Illusion:* The team critiques current popular rankings like Chatbot Arena, discussing how anonymous, unstratified voting can lead to biased results and how companies can game the system.
* *The Xbox Secret to AI Ranking:* Discover how Prolific uses *TrueSkill* —the same algorithm Microsoft developed for Xbox Live matchmaking—to create a fairer, more statistically sound leaderboard for LLMs.
* *The Personality Gap:* Early data from the *Humane Leaderboard* suggests that while AI is getting smarter, it is actually performing *worse* on metrics like personality, culture, and sycophancy (the tendency for models to become annoying people-pleasers).
About the HUMAINE Leaderboard
Moving beyond simple A vs. B testing, the researchers discuss their new framework that samples participants based on *census data* (Age, Ethnicity, Political Alignment). By using a representative sample of the general public rather than just tech enthusiasts, they are building a standard that reflects the values of the real world.
*Are we building models for benchmarks, or are we building them for humans? It’s time to change the scoreboard.*
Rescript link:
https://app.rescript.info/public/share/IDqwjY9Q43S22qSgL5EkWGFymJwZ3SVxvrfpgHZLXQc
TIMESTAMPS:
00:00:00 Introduction & The Benchmarking Problem
00:01:58 The Fractured State of AI Evaluation
00:03:54 AI Safety & Interpretability
00:05:45 Bias in Chatbot Arena
00:06:45 Prolifics Three Pillars Approach
00:09:01 TrueSkill Ranking & Efficient Sampling
00:12:04 Census-Based Representative Sampling
00:13:00 Key Findings: Culture, Personality & Sycophancy
REFERENCES:
Paper:
[00:00:15] MMLU
https://arxiv.org/abs/2009.03300
[00:05:10] Constitutional AI
https://arxiv.org/abs/2212.08073
[00:06:45] The Leaderboard Illusion
https://arxiv.org/abs/2504.20879
[00:09:41] HUMAINE Framework Paper
https://huggingface.co/blog/ProlificAI/humaine-framework
Company:
[00:00:30] Prolific
https://www.prolific.com
[00:01:45] Chatbot Arena
https://lmarena.ai/
Person:
[00:00:35] Andrew Gordon
https://www.linkedin.com/in/andrew-gordon-03879919a/
[00:00:45] Nora Petrova
https://www.linkedin.com/in/nora-petrova/
Event:
Algorithm:
[00:09:01] Microsoft TrueSkill
https://www.microsoft.com/en-us/research/project/trueskill-ranking-system/
Leaderboard:
[00:09:21] Prolific HUMAINE Leaderboard
https://www.prolific.com/humaine
[00:09:31] HUMAINE HuggingFace Space
https://huggingface.co/spaces/ProlificAI/humaine-leaderboard
[00:10:21] Prolific AI Leaderboard Portal
https://www.prolific.com/leaderboard
Dataset:
[00:09:51] Prolific Social Reasoning RLHF Dataset
https://huggingface.co/datasets/ProlificAI/social-reasoning-rlhf
Organization:
[00:10:31] MLCommons
https://mlcommons.org/")
![#68 DR. WALID SABA 2.0 - Natural Language Understanding [UNPLUGGED] @MachineLearningStreetTalk](https://i.ytimg.com/vi/pMtk-iUaEuQ/hqdefault.jpg)
