Uploaded January 2025 | Updated November 2026, 2 hours ago
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.
centml.ai/pricing
2. Sponsorship
[00:03:00] 2.1 TufaAI Labs and CentML
Jeff Clune has spent his career chasing one of science's biggest questions: how did evolution produce the explosion of complexity we see in nature, and can we build algorithms that do the same thing? In this wide-ranging conversation, he lays out the case for open-ended evolutionary algorithms -- systems designed to generate novel and interesting outcomes forever, drawing on principles from both Darwinian evolution and human cultural innovation.
Clune explains the central paradox of his work: trying too hard to accomplish a specific goal is often the worst strategy. Instead, the best results come from recognising serendipity and keeping hold of interestingly new things, regardless of whether they seem immediately useful. This insight, drawn from Kenneth Stanley's work on novelty search, underpins a new generation of algorithms that use foundation models as judges of what counts as genuinely interesting and novel.
The conversation covers POET (evolved environments for reinforcement learning), NEAT (neuroevolution of augmenting topologies), ADAS (automated design of agentic systems), and OMNI-EPIC (using language models to generate open-ended environments). Clune walks through how these systems riff on previous discoveries to create increasingly complex challenges -- from simple ball-kicking tasks through multi-room buildings to cluttered restaurant scenarios that robots must navigate.
The interview also tackles AI safety head-on, with Clune advocating for democratic governance coalitions, regulation of frontier models, and global alignment protocols. He discusses why the interpretability problem may be harder than it looks, how open-ended AI systems could pose unique risks, and his view that the biggest danger is not acting on safety soon enough.
---
REFERENCES:
paper:
[00:02:35] POET: Generating/solving complex challenges
arxiv.org/abs/1901.01753
[00:17:05] Automated capability discovery in foundation models
openreview.net/forum?id=nhgbvyrvTP
[00:18:10] NEAT: NeuroEvolution of Augmenting Topologies
https://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf
[00:26:50] Novelty search vs objective-based optimization
https://www.cs.swarthmore.edu/~meeden/DevelopmentalRobotics/lehman_ecj11.pdf
[00:28:55] AI-generating algorithms approach to AGI
arxiv.org/abs/1905.10985
[00:41:10] Video PreTraining (VPT)
cdn.openai.com/vpt/Paper.pdf
[00:44:00] Thought Cloning: Imitating human thinking
arxiv.org/pdf/2306.00323
[01:15:10] Automated Design of Agentic Systems (ADAS)
arxiv.org/abs/2408.08435
[01:32:30] OMNI-EPIC
arxiv.org/abs/2405.15568
book:
[00:11:10] Why Greatness Cannot Be Planned
amazon.com/Why-Greatness-Cannot-Planned-Objective/dp/3319155237
---
LINKS:
Full Transcript: app.rescript.info/share/1bf7d45e8d7326bba0a73f7fdd686d05
Download PDF transcript: app.rescript.info/api/public/sessions/ceffc76fd4f263da/pdf
Jeff Clune:
https://x.com/jeffclune
jeffclune.com
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.
centml.ai/pricing
2. Sponsorship
[00:03:00] 2.1 TufaAI Labs and CentML
Jeff Clune has spent his career chasing one of science's biggest questions: how did evolution produce the explosion of complexity we see in nature, and can we build algorithms that do the same thing? In this wide-ranging conversation, he lays out the case for open-ended evolutionary algorithms -- systems designed to generate novel and interesting outcomes forever, drawing on principles from both Darwinian evolution and human cultural innovation.
Clune explains the central paradox of his work: trying too hard to accomplish a specific goal is often the worst strategy. Instead, the best results come from recognising serendipity and keeping hold of interestingly new things, regardless of whether they seem immediately useful. This insight, drawn from Kenneth Stanley's work on novelty search, underpins a new generation of algorithms that use foundation models as judges of what counts as genuinely interesting and novel.
The conversation covers POET (evolved environments for reinforcement learning), NEAT (neuroevolution of augmenting topologies), ADAS (automated design of agentic systems), and OMNI-EPIC (using language models to generate open-ended environments). Clune walks through how these systems riff on previous discoveries to create increasingly complex challenges -- from simple ball-kicking tasks through multi-room buildings to cluttered restaurant scenarios that robots must navigate.
The interview also tackles AI safety head-on, with Clune advocating for democratic governance coalitions, regulation of frontier models, and global alignment protocols. He discusses why the interpretability problem may be harder than it looks, how open-ended AI systems could pose unique risks, and his view that the biggest danger is not acting on safety soon enough.
---
REFERENCES:
paper:
[00:02:35] POET: Generating/solving complex challenges
arxiv.org/abs/1901.01753
[00:17:05] Automated capability discovery in foundation models
openreview.net/forum?id=nhgbvyrvTP
[00:18:10] NEAT: NeuroEvolution of Augmenting Topologies
https://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf
[00:26:50] Novelty search vs objective-based optimization
https://www.cs.swarthmore.edu/~meeden/DevelopmentalRobotics/lehman_ecj11.pdf
[00:28:55] AI-generating algorithms approach to AGI
arxiv.org/abs/1905.10985
[00:41:10] Video PreTraining (VPT)
cdn.openai.com/vpt/Paper.pdf
[00:44:00] Thought Cloning: Imitating human thinking
arxiv.org/pdf/2306.00323
[01:15:10] Automated Design of Agentic Systems (ADAS)
arxiv.org/abs/2408.08435
[01:32:30] OMNI-EPIC
arxiv.org/abs/2405.15568
book:
[00:11:10] Why Greatness Cannot Be Planned
amazon.com/Why-Greatness-Cannot-Planned-Objective/dp/3319155237
---
LINKS:
Full Transcript: app.rescript.info/share/1bf7d45e8d7326bba0a73f7fdd686d05
Download PDF transcript: app.rescript.info/api/public/sessions/ceffc76fd4f263da/pdf
Jeff Clune:
https://x.com/jeffclune
jeffclune.com
![How Researchers Test AI for Hidden Goals — Apollo Research
Can an AI do the right thing for the wrong reason? Tim Scarfe speaks with Apollo Research’s Alexander Meinke, Axel Højmark and Jérémy Scheurer about Measuring Reward-Seeking via Contrastive Belief Updates, their new research with OpenAI.
The panel asks how models infer what graders reward, why good behaviour can come from the wrong reason, and whether that difference can be measured. The conversation moves through promise-breaking, grader awareness, reward hacking, scheming, opaque reasoning and corrigibility, then turns to a detailed walkthrough of the contrastive-belief method and what its results do and do not show. The o3 results discussed here concern an intermediate checkpoint without safety training.
This episode was made in partnership with Apollo Research. MLST retained full editorial control.
Reference
Apollo Research: https://www.apolloresearch.ai/
TIMESTAMPS:
00:00:00 Cold Open
00:02:12 Right Things, Wrong Reasons
00:12:47 Grader Awareness
00:26:22 Legibility
00:32:35 What To Call It
00:35:58 Intelligence, Agency, Anthropomorphism
00:45:16 Apollo’s Mission
00:48:54 The End of the Exponential
00:55:45 The Paper
01:16:34 Closing Reflection
REFERENCES:
tool:
[00:00:08] Claude Fable
https://www.anthropic.com/claude/fable
[00:12:50] AlphaGo Zero
https://deepmind.google/blog/alphago-zero-starting-from-scratch/
[00:44:30] AlphaFold 3
https://deepmind.google/science/alphafold/
paper:
[00:01:02] Measuring Reward-Seeking via Contrastive Belief Updates
https://arxiv.org/abs/2607.18966
[00:16:19] Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
https://transformer-circuits.pub/2026/nla/
[00:26:48] Stress Testing Deliberative Alignment for Anti-Scheming Training
https://arxiv.org/abs/2509.15541
[00:35:33] Shortcut learning in deep neural networks
https://arxiv.org/abs/2004.07780
[00:53:49] Measuring AI Ability to Complete Long Software Tasks
https://arxiv.org/abs/2503.14499
[00:59:52] Modifying LLM Beliefs with Synthetic Document Finetuning
https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/
[01:10:44] Alignment Faking in Large Language Models
https://arxiv.org/abs/2412.14093
[01:13:55] Natural Emergent Misalignment from Reward Hacking
https://www.anthropic.com/research/emergent-misalignment-reward-hacking
other:
[00:10:14] We Need a Science of Scheming
https://www.apolloresearch.ai/science/science-of-scheming/
[00:32:56] CoastRunners reward hacking example
https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/
organization:
[01:06:07] Redwood Research
https://www.redwoodresearch.org/
ReScript:
https://app.rescript.info/share/718ab68e18cfa3b9b800da6b3290fd42 How Researchers Test AI for Hidden Goals — Apollo Research](https://i.ytimg.com/vi/n1Qk8xbqF-M/mqdefault.jpg)
![Language Models are Modelling The World [Nicholas Carlini]
Nicholas Carlini from Google DeepMind offers his view of AI security, emergent LLM capabilities, and his groundbreaking model-stealing research. He reveals how LLMs can unexpectedly excel at tasks like chess and discusses the security pitfalls of LLM-generated code.
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.
https://centml.ai/pricing/
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events?
Goto https://tufalabs.ai/
***
Transcript: https://www.dropbox.com/scl/fi/lat7sfyd4k3g5k9crjpbf/CARLINI.pdf?rlkey=b7kcqbvau17uw6rksbr8ccd8v&dl=0
TOC:
1. ML Security Fundamentals
[00:00:00] 1.1 ML Model Reasoning and Security Fundamentals
[00:03:04] 1.2 ML Security Vulnerabilities and System Design
[00:08:22] 1.3 LLM Chess Capabilities and Emergent Behavior
[00:13:20] 1.4 Model Training, RLHF, and Calibration Effects
2. Model Evaluation and Research Methods
[00:19:40] 2.1 Model Reasoning and Evaluation Metrics
[00:24:37] 2.2 Security Research Philosophy and Methodology
[00:27:50] 2.3 Security Disclosure Norms and Community Differences
3. LLM Applications and Best Practices
[00:44:29] 3.1 Practical LLM Applications and Productivity Gains
[00:49:51] 3.2 Effective LLM Usage and Prompting Strategies
[00:53:03] 3.3 Security Vulnerabilities in LLM-Generated Code
4. Advanced LLM Research and Architecture
[00:59:13] 4.1 LLM Code Generation Performance and O(1) Labs Experience
[01:03:31] 4.2 Adaptation Patterns and Benchmarking Challenges
[01:10:10] 4.3 Model Stealing Research and Production LLM Architecture Extraction
REFS:
[00:01:15] Nicholas Carlini’s personal website & research profile (Google DeepMind, ML security) - https://nicholas.carlini.com/
[00:01:50] CentML AI compute platform for language model workloads - https://centml.ai/
[00:04:30] Seminal paper on neural network robustness against adversarial examples (Carlini & Wagner, 2016) - https://arxiv.org/abs/1608.04644
[00:05:20] Computer Fraud and Abuse Act (CFAA) – primary U.S. federal law on computer hacking liability - https://www.justice.gov/jm/jm-9-48000-computer-fraud
[00:08:30] Blog post: Emergent chess capabilities in GPT-3.5-turbo-instruct (Nicholas Carlini, Sept 2023) - https://nicholas.carlini.com/writing/2023/chess-llm.html
[00:16:10] Paper: “Self-Play Preference Optimization for Language Model Alignment” (Yue Wu et al., 2024) - https://arxiv.org/abs/2405.00675
[00:18:00] GPT-4 Technical Report: development, capabilities, and calibration analysis - https://arxiv.org/abs/2303.08774
[00:22:40] Historical shift from descriptive to algebraic chess notation (FIDE) - https://en.wikipedia.org/wiki/Descriptive_notation
[00:23:55] Analysis of distribution shift in ML (Hendrycks et al.) - https://arxiv.org/abs/2006.16241
[00:27:40] Nicholas Carlini’s essay “Why I Attack” (June 2024) – motivations for security research - https://nicholas.carlini.com/writing/2024/why-i-attack.html
[00:34:05] Google Project Zero’s 90-day vulnerability disclosure policy - https://googleprojectzero.blogspot.com/p/vulnerability-disclosure-policy.html
[00:51:15] Evolution of Google search syntax & user behavior (Daniel M. Russell) - https://www.amazon.com/Joy-Search-Google-Master-Information/dp/0262042878
[01:04:05] Rust’s ownership & borrowing system for memory safety - https://doc.rust-lang.org/book/ch04-00-understanding-ownership.html
[01:10:05] Paper: “Stealing Part of a Production Language Model” (Carlini et al., March 2024) – extraction attacks on ChatGPT, PaLM-2 - https://arxiv.org/abs/2403.06634
[01:10:55] First model stealing paper (Tramèr et al., 2016) – attacking ML APIs via prediction - https://arxiv.org/abs/1609.02943 Language Models are Modelling The World [Nicholas Carlini]](https://i.ytimg.com/vi/n4ipEJ6uJ44/mqdefault.jpg)



![Why does the Chinese Room still haunt AI?
Keith Duggar and Tim Scarfe return for the second edition of their hosts-only philosophical steakhouse, picking up right where the last episode left off with Keith fresh from his appearance on Liron Shapiras Doom Debates show. The conversation starts technical and stays there for a good while, working through a precise argument about why standard autoregressive LLMs are not Turing complete in the technical sense.
Keith lays out the distinction between potentially infinite and actually infinite memory, explains why the algorithms learned through gradient descent on fixed-context-window architectures come from the finite state automata class rather than the Turing machine class, and confronts the practical objection head-on: even if this is technically true, does it matter? Tim pushes back with RAG systems that expand effective memory, and the two work through whether closing the read-write loop could unlock a richer algorithm space.
The middle hour is a sustained engagement with John Searles Chinese Room argument. They dig into the gap between syntax and semantics, why gears churning out responses lack something even if they pass behavioral tests, and what Searle means by requiring the right causal structure for understanding. Bishops Dancing with Pixies reductio, Wolframs computational boundedness, and Fristons temporal-counterfactual depth model of self-awareness all get pulled into the conversation. Keith makes one of the more interesting concessions: he cannot rule out that a sufficiently complex program with the right causal structure could cross the threshold into genuine understanding.
The final act covers the Nobel Prize controversy around deep learning (Keith declines to judge but notes academias political problems), Chomskys critique that neural networks are not scientific theories, and a framework Keith developed on Doom Debates: focusing on AI harm rather than AI doom as a way to build broader policy coalitions. Tim distinguishes AI ethics from AI safety and both agree the speculative existential arguments, while philosophically legitimate, should not drive policy at the same level as the concrete social harms already visible.
Recorded Friday 11th October 2024.
REFERENCES:
video:
[00:00:30] Keith Duggar on Doom Debates
https://www.youtube.com/watch?v=4v-Qh3JQ4Jc
[00:00:30] Is o1 Reasoning? (MLST Philosophical Steakhouse 1)
https://www.youtube.com/watch?v=nO6sDk6vO0g
[00:55:00] J. Mark Bishop on MLST
https://www.youtube.com/watch?v KVAzAzO5HU
[00:55:10] Searle Google Talk
https://www.youtube.com/watch?v=rHKwIYsPXLg
paper:
[00:10:50] On the Measure of Intelligence
https://arxiv.org/abs/1911.01547
[00:31:40] Minds, Brains, and Programs
https://home.csulb.edu/~cwallis/382/readings/482/searle.minds.brains.programs.bbs.1980.pdf
[00:55:00] Dancing with Pixies
https://philarchive.org/rec/BISDWP-2
[00:55:00] Artificial Intelligence Is Stupid and Causal Reasoning Will Not Fix It
https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2020.513474/full
[00:58:04] Nestedly Recursive Functions
https://writings.stephenwolfram.com/2024/09/nestedly-recursive-functions/
[01:33:00] Deconstructing the AI Myth: Fallacies and Harms of Algorithmification
https://www.researchgate.net/publication/382802495_Deconstructing_the_AI_Myth_Fallacies_and_Harms_of_Algorithmification
[01:33:30] What Is the Philosophy of Information
https://www.researchgate.net/publication/225070243_What_Is_the_Philosophy_of_Information
concept:
[00:31:40] Chinese Room Argument
https://plato.stanford.edu/entries/chinese-room/
book:
[00:58:04] Godel, Escher, Bach: An Eternal Golden Braid
https://www.amazon.co.uk/Godel-Escher-Bach-Eternal-Golden/dp/0465026567
[01:22:20] Principles of Deep Learning Theory
https://www.amazon.com/Principles-Deep-Learning-Theory-Science/dp/1316519333
LINKS:
Full Transcript: https://app.rescript.info/share/e6d0ea728cf3b83bbabf3e0bdf52e036
Download PDF transcript: https://app.rescript.info/api/public/sessions/68b8550291209502/pdf Why does the Chinese Room still haunt AI?](https://i.ytimg.com/vi/nnBbpPt2SKI/mqdefault.jpg)
![Why Every AI Model Is an Impostor — Kenneth Stanley
What if todays incredible AI is just a brilliant impostor?
This episode features host Dr. Tim Scarfe in conversation with guests Prof. Kenneth Stanley (ex-OpenAI), Dr. Keith Duggar (MIT), and Akarsh Kumar (MIT).
While AI today produces amazing results on the surface, its internal understanding is a complete mess, described as total spaghetti [00:00:49]. This is because its trained with a brute-force method (SGD) that’s like building a sandcastle: it looks right from a distance, but has no real structure holding it together [00:01:45].
To explain the difference, Keith Duggar shares a great analogy about his high school physics classes [00:03:18]. One class was about memorizing lots of formulas for specific situations (like the impostor AI). The other used calculus to derive the answers from a deeper understanding, which was much easier and more powerful. This is the core difference: one method memorizes, the other truly understands.
The episode then introduces a different, more powerful way to build AI, based on Kenneth Stanleys old experiment, Picbreeder [00:04:45]. This method creates AI with a shockingly clean and intuitive internal model of the world. For example, it might develop a model of a skull where it understands the mouth as a separate component it can open and close, without ever being explicitly trained on that action [00:06:15]. This deep understanding emerges bottom-up, without massive datasets.
The secret is to abandon a fixed goal and embrace deception [00:08:42]—the idea that the stepping stones to a great discovery often dont look anything like the final result. Instead of optimizing for a target, the AI is built through an open-ended process of exploring whats interesting [00:09:15]. This creates a more flexible and adaptable foundation, a bit like how evolvability wins out in nature [00:10:30].
The show concludes by arguing that this choice matters immensely. The impostor path may be hitting a wall, requiring insane amounts of money and energy for progress and failing to deliver true creativity or continual learning [00:13:00]. The ultimate message is a call to not put all our eggs in one basket [00:14:25]. We should explore these open-ended, creative paths to discover a more genuine form of intelligence, which may be found where we least expect it.
Extended interview here: https://www.youtube.com/watch?v=KKUKikuV58o
REFS:
Questioning Representational Optimism in Deep Learning:
The Fractured Entangled Representation Hypothesis
Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley
https://arxiv.org/pdf/2505.11581
Kenneth O. Stanley, Joel Lehman
Why Greatness Cannot Be Planned: The Myth of the Objective
https://amzn.to/44xLaXK
Original show with Kenneth from 4 years ago:
https://www.youtube.com/watch?v=lhYGXYeMq_E
Kenneth Stanley is SVP Open Endedness at Lila Sciences
https://x.com/kenneth0stanley
Akarsh Kumar (MIT)
https://akarshkumar.com/
AND... Kenneth is HIRING (this is an OPPORTUNITY OF A LIFETIME!)
Research Engineer: https://job-boards.greenhouse.io/lila/jobs/7890007002
Research Scientist: https://job-boards.greenhouse.io/lila/jobs/8012245002
Tims Code visualisation of FER based on Akarsh repo: https://github.com/ecsplendid/fer
TRANSCRIPT: https://app.rescript.info/public/share/YKAZzZ6lwZkjTLRpVJreOOxGhLI8y4m3fAyU8NSavx0 Why Every AI Model Is an Impostor — Kenneth Stanley](https://i.ytimg.com/vi/o1q6Hhz0MAg/mqdefault.jpg)
![Taming Silicon Valley - Prof. Gary Marcus
Gary Marcus, cognitive scientist and one of AIs most prominent critics, sits down with Tim Scarfe for a nearly two-hour dissection of the current state of artificial intelligence and the tech industry that builds it. Timed to the release of his book Taming Silicon Valley, Marcus makes a sustained case that large language models remain fundamentally brittle — impressive on the surface, yet lacking the compositional semantics, object permanence, and genuine world models that would make them reliable.
The discussion ranges across AI safety theater and the gap between marketing claims and actual capability, the persistent failure modes of image generation and code synthesis, and why LLMs playing chess badly tells us something important about what they have and have not learned. Marcus is especially pointed on industry resistance to regulation, walking through the politics of Californias SB-1047, the EU AI Act, and the structural reasons why voluntary self-governance by tech companies has not worked.
The second half turns to the societal damage already visible: copyright battles, the firehose of AI-generated misinformation, surveillance capitalism, and the erosion of trust in digital information. Marcus and Scarfe spar over x-risk and alignment, with Marcus arguing that near-term harms deserve more attention than speculative doom scenarios. The conversation closes on the prospects for neuro-symbolic AI, the legacy of thinkers like Chomsky and Piaget, and what it would actually take to build systems that reason rather than pattern-match.
REFERENCES:
book:
[00:00:00] Gary Marcus - Taming Silicon Valley
https://amzn.to/3XTlC5s
[00:57:26] Shoshana Zuboff - Surveillance Capitalism
https://amzn.to/3ZqHAxS
[01:23:14] Sayash Kapoor and Arvind Narayanan - AI Snake Oil
https://www.aisnakeoil.com/
[01:23:14] Isaac Asimov - Three Laws of Robotics
https://amzn.to/3XTIwtl
[01:44:33] Gary Marcus - The Algebraic Mind
https://mitpress.mit.edu/books/algebraic-mind
person:
[00:00:00] Gary Marcus Substack
https://garymarcus.substack.com/
[00:23:49] Jean Piaget - Object Permanence Theory
https://en.wikipedia.org/wiki/Object_permanence
[01:44:33] Seymour Papert - Logo Programming Language
https://el.media.mit.edu/logo-foundation/what_is_logo/logo_primer.html
paper:
[00:23:49] Evelina Leivada et al. - LLMs Understanding Human Language
https://arxiv.org/pdf/2308.00109
[00:31:09] Alan Turing - Computing Machinery and Intelligence
https://academic.oup.com/mind/article/LIX/236/433/986238
[00:34:45] Nicholas Carlini - Chess with LLMs
https://nicholas.carlini.com/writing/2023/chess-llm.html
[00:34:45] Mathieu Acher - GPT-4 Chess Analysis
https://blog.mathieuacher.com/ChessWinning7MovesGPT/
[00:42:10] Jiexin Wang - AI-Assisted Coding Security
https://arxiv.org/pdf/2407.02395v1
[00:42:10] Rodney Brooks - Three Laws of AI
https://rodneybrooks.com/rodney-brooks-three-laws-of-artificial-intelligence/
[00:48:10] Gary Marcus - Open Letter on SB-1047
https://garymarcus.substack.com/p/an-open-letter-to-fei-fei-li-concerning
[00:57:26] Jaron Lanier - Twitter Poisoning
https://www.nytimes.com/2022/11/11/opinion/trump-musk-kanye-twitter.html
[00:57:26] Chris Lu et al. - AI Scientist Paper
https://arxiv.org/abs/2408.06292
[01:23:14] Gary Marcus - p(doom) Analysis
https://garymarcus.substack.com/p/d28
[01:44:33] Gary Marcus - Neural Networks and Generalization
https://www.sciencedirect.com/science/article/pii/S0010028598906946
other:
[00:23:49] Gottlob Frege - Compositional Semantics
https://plato.stanford.edu/entries/compositionality/
[00:42:10] Cruise Teleoperation Revelation
https://www.nytimes.com/2023/11/03/technology/cruise-general-motors-self-driving-cars.html
[00:48:10] California SB-1047 AI Regulation
https://apcp.assembly.ca.gov/system/files/2024-06/sb-1047-wiener-apcp-analysis_0.pdf
[00:48:10] European Commission - EU AI Act
https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
[00:54:55] A&M Records v. Napster - Copyright Precedent
https://en.wikipedia.org/wiki/A%26M_Records,_Inc._v._Napster,_Inc.
[00:57:26] Firehose of Falsehood - Russian Propaganda Model
https://en.wikipedia.org/wiki/Firehose_of_falsehood
[01:44:33] Laplace Demon Concept
https://en.wikipedia.org/wiki/Laplace%27s_demon
video:
[00:57:26] Adam Curtis - HyperNormalisation
https://www.imdb.com/title/tt6156350/
[00:57:26] Luciano Floridi - Digital Ethics
https://www.youtube.com/watch?v=YLNGvvgq3eg
[01:44:33] MLST - Noam Chomsky Interview
https://www.youtube.com/watch?v=axuGfh4UR9Q
[01:44:33] Beff Jezos - Physics-Inspired Intelligence
https://www.youtube.com/watch?v=0zxi0xSBOaQ
LINKS:
Full Transcript: https://app.rescript.info/share/b2f49ea9ca469401e8f65b29156495c8
Download PDF transcript: https://app.rescript.info/api/public/sessions/530cbbb31f702d43/pdf
Gary Marcus:
https://garymarcus.substack.com/
https://x.com/GaryMarcus Taming Silicon Valley - Prof. Gary Marcus](https://i.ytimg.com/vi/o9MfuUoGlSw/mqdefault.jpg)
![Why Program Synthesis Is Next (Kevin Ellis and Zenna Tavares)
Kevin Ellis (Cornell) and Zenna Tavares (BASIS) argue that the next wave of AI needs to learn like humans do: building abstract models from small amounts of data through active exploration, not just passive pattern matching at scale.
The conversation centers on their joint work comparing two fundamentally different ways of solving problems. Induction searches for an explicit program something you could write in Python that transforms inputs to outputs. Transduction skips the program and directly predicts the answer, the way a neural network would. On the Abstraction and Reasoning Corpus (ARC), these approaches turn out to be complementary: some problems yield to systematic symbolic search, others to neural intuition. The ensemble is stronger than either alone, and the reasons connect to findings in cognitive science about when explicit reasoning helps versus hurts.
Kevin explains how his DreamCoder work pioneered a wake-sleep cycle for program synthesis: dream up programs, run them to see what they do, learn the inverse mapping, then wake up and let real-world failures adjust the distribution of dreams. The modern version replaces explicit symbolic libraries with in-context learning over LLM-generated code, keeping the same iterative refinement loop.
Zenna introduces his Autumn system for synthesizing the source code of interactive environments from observed behavior a form of computational science where the model must also infer hidden state it cannot directly observe. Both researchers converge on the idea that abstraction is the key unsolved problem: real intelligence requires knowing what to ignore, not just what to represent. Zenna frames this through resource rationality choosing the right level of abstraction given your computational budget and expected tasks.
The discussion closes with Project MARA, their joint effort to build interactive benchmarks that go beyond ARCs static puzzles, requiring agents to actively explore and build world models from scratch.
SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.
REFERENCES:
Paper:
[00:00:25] DreamCoder: Growing Generalizable, Interpretable Knowledge with Wake-Sleep Bayesian Program Learning
https://arxiv.org/abs/2006.08381
[00:01:10] Mind Your Step: Active Search over Compositional Spaces
https://arxiv.org/abs/2410.21333
[00:06:05] Bayesian inference in the cognitive sciences
https://psycnet.apa.org/record/2008-06911-003
[00:13:00] Induction and Transduction
https://arxiv.org/abs/2411.02272
[00:23:15] Neurosymbolic AI: The 3rd Wave
https://arxiv.org/abs/2012.05876
[00:38:35] On the Measure of Intelligence (ARC)
https://arxiv.org/abs/1911.01547
[00:39:20] Causal Reactive Programs (Autumn)
http://www.zenna.org/publications/autumn2022.pdf
[00:42:50] MuZero
http://arxiv.org/pdf/1911.08265
[00:43:20] VisualPredicator
https://arxiv.org/abs/2410.23156
Book:
[00:48:55] Bayesian Models of Cognition
https://mitpress.mit.edu/9780262049412/bayesian-models-of-cognition/
Essay:
[00:49:30] The Bitter Lesson
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Project:
[01:11:55] Project MARA
https://www.basis.ai/blog/mara/
LINKS:
Full Transcript: https://app.rescript.info/share/e0a208e545cabae728a3d72f76fcd310
Download PDF transcript: https://app.rescript.info/api/public/sessions/f47975e800b064d9/pdf Why Program Synthesis Is Next (Kevin Ellis and Zenna Tavares)](https://i.ytimg.com/vi/oYTm0p3DCzg/mqdefault.jpg)
![Why Every Brain Metaphor in History Has Been Wrong [SPECIAL EDITION]
What if everything we think we know about the brain is just a really good metaphor that we forgot was a metaphor?
This episode takes you on a journey through the history of scientific simplification, from a young Karl Friston watching wood lice in his garden to the bold claims that your mind is literally software running on biological hardware.
We bring together some of the most brilliant minds weve interviewed — Professor Mazviita Chirimuuta, Francois Chollet, Joscha Bach, Professor Luciano Floridi, Professor Noam Chomsky, Nobel laureate John Jumper, and more — to wrestle with a deceptively simple question: *When scientists simplify reality to study it, what gets captured and what gets lost?*
*Key ideas explored:*
*The Spherical Cow Problem* — Science requires simplification. Were limited creatures trying to understand systems far more complex than our working memory can hold. But when does a useful model become a dangerous illusion?
*The Kaleidoscope Hypothesis* — Francois Chollets beautiful idea that beneath all the apparent chaos of reality lies simple, repeating patterns — like bits of colored glass in a kaleidoscope creating infinite complexity. Is this profound truth or Platonic wishful thinking?
*Is Software Really Spirit?* — Joscha Bach makes the provocative claim that software is literally spirit, not metaphorically. We push back hard on this, asking whether the sameness we see across different computers running the same program exists in nature or only in our descriptions.
*The Cultural Illusion of AGI* — Why does artificial general intelligence seem so inevitable to people in Silicon Valley? Professor Chirimuuta suggests we might be caught in a cultural historical illusion — our mechanistic assumptions about minds making AI seem like destiny when it might just be a bet.
*Prediction vs. Understanding* — Nobel Prize winner John Jumper: AI can predict and control, but understanding requires a human in the loop.
Throughout history, weve described the brain as hydraulic pumps, telegraph networks, telephone switchboards, and now computers. Each metaphor felt obviously true at the time. This episode asks: what will we think was naive about our current assumptions in fifty years?
Featuring insights from *The Brain Abstracted* by Mazviita Chirimuuta — possibly the most influential book on how we think about thinking in 2025.
TIMESTAMPS:
00:00:00 The Wood Louse & The Spherical Cow
00:02:04 The Necessity of Abstraction
00:04:42 Simplicius vs. Ignorantio: The Boxing Match
00:06:39 The Kaleidoscope Hypothesis
00:08:40 Is the Mind Software?
00:13:15 Critique of Causal Patterns
00:14:40 Temperature is Not a Thing
00:18:24 The Ship of Theseus & Ontology
00:23:45 Metaphors Hardening into Reality
00:25:41 The Illusion of AGI Inevitability
00:27:45 Prediction vs. Understanding
00:32:00 Climbing the Mountain vs. The Helicopter
00:34:53 Haptic Realism & The Limits of Knowledge
REFERENCES:
Person:
[00:00:00] Karl Friston (UCL)
https://profiles.ucl.ac.uk/1236-karl-friston
[00:06:30] Francois Chollet
https://fchollet.com/
[00:14:41] Cesar Hidalgo, MLST interview.
https://www.youtube.com/watch?v=vzpFOJRteeI
[00:30:30] Terence Taos Blog
https://terrytao.wordpress.com/
Book:
[00:02:25] The Brain Abstracted
https://mitpress.mit.edu/9780262548045/the-brain-abstracted/
[00:06:00] On Learned Ignorance
https://www.amazon.com/Nicholas-Cusa-learned-ignorance-translation/dp/0938060236
[00:24:15] Science and the Modern World
https://amazon.com/dp/0684836394
Interview.:
[00:02:43] The Brain Abstracted Patreon interview
https://www.patreon.com/posts/brain-abstracted-124479979
Interview:
[00:04:18] David Krakauers presentation on intelligence.
https://www.youtube.com/watch?v=dY46YsGWMIc
[00:06:45] Machine Learning Street Talk interview with Francois Chollet.
https://www.youtube.com/watch?v=JTU8Ha4Jyfc
[00:09:32] Joscha Bach Patreon interview.
https://www.patreon.com/posts/joscha-bach-deep-141884561
[00:18:24] Luciano Floridi, MLST interview.
https://www.youtube.com/watch?v=YLNGvvgq3eg
[00:25:02] Jeff Beck, MLST interview.
https://www.youtube.com/watch?v=9suqiofCiwM
[00:28:00] John Jumper
https://www.patreon.com/posts/john-jumper-on-144557652
[00:30:48] Noam Chomsky, MLST interview.
https://www.youtube.com/watch?v=axuGfh4UR9Q
[00:31:59] Anna Ciaunica
https://www.patreon.com/posts/dr-anna-ciaunica-144509970
[00:33:12] Mike Israetel debate on functionalism.
https://www.youtube.com/watch?v=4yYcN_mFi18
Company/Org:
[00:09:25] Neuralink
https://neuralink.com/
Paper:
[00:23:45] A Logical Calculus of Ideas Immanent in Nervous Activity
https://link.springer.com/article/10.1007/BF02478259
[00:28:00] Highly Accurate Protein Structure Prediction with AlphaFold
https://www.nature.com/articles/s41586-021-03819-2
Thank you to Dr. Maxwell Ramstead for early script work on this show (Ph.D student of Friston) and the woodlice story came from him! Why Every Brain Metaphor in History Has Been Wrong [SPECIAL EDITION]](https://i.ytimg.com/vi/pO0WZsN8Oiw/mqdefault.jpg)
