Why does the Chinese Room still haunt AI? @MachineLearningStreetTalk
Why does the Chinese Room still haunt AI?  @MachineLearningStreetTalk
Uploaded October 2024 | Updated September 2026, 1 week ago
Keith Duggar and Tim Scarfe return for the second edition of their hosts-only philosophical steakhouse, picking up right where the last episode left off -- with Keith fresh from his appearance on Liron Shapira's Doom Debates show. The conversation starts technical and stays there for a good while, working through a precise argument about why standard autoregressive LLMs are not Turing complete in the technical sense.

Keith lays out the distinction between potentially infinite and actually infinite memory, explains why the algorithms learned through gradient descent on fixed-context-window architectures come from the finite state automata class rather than the Turing machine class, and confronts the practical objection head-on: even if this is technically true, does it matter? Tim pushes back with RAG systems that expand effective memory, and the two work through whether closing the read-write loop could unlock a richer algorithm space.

The middle hour is a sustained engagement with John Searle's Chinese Room argument. They dig into the gap between syntax and semantics, why gears churning out responses lack something even if they pass behavioral tests, and what Searle means by requiring the right causal structure for understanding. Bishop's Dancing with Pixies reductio, Wolfram's computational boundedness, and Friston's temporal-counterfactual depth model of self-awareness all get pulled into the conversation. Keith makes one of the more interesting concessions: he cannot rule out that a sufficiently complex program with the right causal structure could cross the threshold into genuine understanding.

The final act covers the Nobel Prize controversy around deep learning (Keith declines to judge but notes academia's political problems), Chomsky's critique that neural networks are not scientific theories, and a framework Keith developed on Doom Debates: focusing on AI harm rather than AI doom as a way to build broader policy coalitions. Tim distinguishes AI ethics from AI safety and both agree the speculative existential arguments, while philosophically legitimate, should not drive policy at the same level as the concrete social harms already visible.

Recorded Friday 11th October 2024.

---
REFERENCES:
video:
[00:00:30] Keith Duggar on Doom Debates
youtube.com/watch?v=4v-Qh3JQ4Jc
[00:00:30] Is o1 Reasoning? (MLST Philosophical Steakhouse 1)
youtube.com/watch?v=nO6sDk6vO0g
[00:55:00] J. Mark Bishop on MLST
youtube.com/watch?v=_KVAzAzO5HU
[00:55:10] Searle Google Talk
youtube.com/watch?v=rHKwIYsPXLg
paper:
[00:10:50] On the Measure of Intelligence
arxiv.org/abs/1911.01547
[00:31:40] Minds, Brains, and Programs
https://home.csulb.edu/~cwallis/382/readings/482/searle.minds.brains.programs.bbs.1980.pdf
[00:55:00] Dancing with Pixies
philarchive.org/rec/BISDWP-2
[00:55:00] Artificial Intelligence Is Stupid and Causal Reasoning Will Not Fix It
frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2020.513474/full
[00:58:04] Nestedly Recursive Functions
writings.stephenwolfram.com/2024/09/nestedly-recursive-functions
[01:33:00] Deconstructing the AI Myth: Fallacies and Harms of Algorithmification
researchgate.net/publication/382802495_Deconstructing_the_AI_Myth_Fallacies_and_Harms_of_Algorithmification
[01:33:30] What Is the Philosophy of Information
researchgate.net/publication/225070243_What_Is_the_Philosophy_of_Information
concept:
[00:31:40] Chinese Room Argument
https://plato.stanford.edu/entries/chinese-room/
book:
[00:58:04] Godel, Escher, Bach: An Eternal Golden Braid
amazon.co.uk/Godel-Escher-Bach-Eternal-Golden/dp/0465026567
[01:22:20] Principles of Deep Learning Theory
amazon.com/Principles-Deep-Learning-Theory-Science/dp/1316519333

---
LINKS:
Full Transcript: app.rescript.info/share/e6d0ea728cf3b83bbabf3e0bdf52e036
Download PDF transcript: app.rescript.info/api/public/sessions/68b8550291209502/pdf
Why does the Chinese Room still haunt AI?Why Every AI Model Is an Impostor — Kenneth StanleyTaming Silicon Valley - Prof. Gary MarcusWhy Program Synthesis Is Next (Kevin Ellis and Zenna Tavares)#70 - LETITIA PARCALABESCU - Symbolics, Linguistics [UNPLUGGED]Why Every Brain Metaphor in History Has Been Wrong [SPECIAL EDITION]Prof. Kenneth Stanley on Creativity and SerendipityGoogle Researcher Shows Life Emerges From Code [Blaise Agüera y Arcas]AI Is Learning at the Wrong Level of Abstraction — Matthieu WyartWE MUST ADD STRUCTURE TO DEEP LEARNING BECAUSE...Why High Benchmark Scores Don’t Mean Better AI [SPONSORED]Building a GENERAL AI agent with reinforcement learning
Machine Learning Street Talk |

Why does the Chinese Room still haunt AI?

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER