Uploaded August 2024 | Updated September 2026, 1 week ago
Jürgen Schmidhuber, the father of generative AI shares his groundbreaking work in deep learning and artificial intelligence. In this exclusive interview, he discusses the history of AI, some of his contributions to the field, and his vision for the future of intelligent machines. Schmidhuber offers unique insights into the exponential growth of technology and the potential impact of AI on humanity and the universe.
MLST is sponsored by Brave:
The Brave Search API covers over 20 billion webpages, built from scratch without Big Tech biases or the recent extortionate price hikes on search API access. Perfect for AI model training and retrieval augmentated generation. Try it now - get 2,000 free queries monthly at brave.com/api.
TOC
00:00:00 Intro
00:03:38 Reasoning
00:13:09 Potential AI Breakthroughs Reducing Computation Needs
00:20:39 Memorization vs. Generalization in AI
00:25:19 Approach to the ARC Challenge
00:29:10 Perceptions of Chat GPT and AGI
00:58:45 Abstract Principles of Jurgen's Approach
01:04:17 Analogical Reasoning and Compression
01:05:48 Breakthroughs in 1991: the P, the G, and the T in ChatGPT and Generative AI
01:15:50 Use of LSTM in Language Models by Tech Giants
01:21:08 Neural Network Aspect Ratio Theory
01:26:53 Reinforcement Learning Without Explicit Teachers
Refs:
★ "Annotated History of Modern AI and Deep Learning" (2022 survey by Schmidhuber):
★ Chain Rule For Backward Credit Assignment (Leibniz, 1676)
★ First Neural Net / Linear Regression / Shallow Learning (Gauss & Legendre, circa 1800)
★ First 20th Century Pioneer of Practical AI (Quevedo, 1914)
★ First Recurrent NN (RNN) Architecture (Lenz, Ising, 1920-1925)
★ AI Theory: Fundamental Limitations of Computation and Computation-Based AI (Gödel, 1931-34)
★ Unpublished ideas about evolving RNNs (Turing, 1948)
★ Multilayer Feedforward NN Without Deep Learning (Rosenblatt, 1958)
★ First Published Learning RNNs (Amari and others, ~1972)
★ First Deep Learning (Ivakhnenko & Lapa, 1965)
★ Deep Learning by Stochastic Gradient Descent (Amari, 1967-68)
★ ReLUs (Fukushima, 1969)
★ Backpropagation (Linnainmaa, 1970); precursor (Kelley, 1960)
★ Backpropagation for NNs (Werbos, 1982)
★ First Deep Convolutional NN (Fukushima, 1979); later combined with Backprop (Waibel 1987, Zhang 1988).
★ Metalearning or Learning to Learn (Schmidhuber, 1987)
★ Generative Adversarial Networks / Artificial Curiosity / NN Online Planners (Schmidhuber, Feb 1990; see the G in Generative AI and ChatGPT)
★ NNs Learn to Generate Subgoals and Work on Command (Schmidhuber, April 1990)
★ NNs Learn to Program NNs: Unnormalized Linear Transformer (Schmidhuber, March 1991; see the T in ChatGPT)
★ Deep Learning by Self-Supervised Pre-Training. Distilling NNs (Schmidhuber, April 1991; see the P in ChatGPT)
★ Experiments with Pre-Training; Analysis of Vanishing/Exploding Gradients, Roots of Long Short-Term Memory / Highway Nets / ResNets (Hochreiter, June 1991, further developed 1999-2015 with other students of Schmidhuber)
★ LSTM journal paper (1997, most cited AI paper of the 20th century)
★ xLSTM (Hochreiter, 2024)
★ Reinforcement Learning Prompt Engineer for Abstract Reasoning and Planning (Schmidhuber 2015)
★ Mindstorms in Natural Language-Based Societies of Mind (2023 paper by Schmidhuber's team)
arxiv.org/abs/2305.17066
★ Bremermann's physical limit of computation (1982)
EXTERNAL LINKS
CogX 2018 - Professor Juergen Schmidhuber
youtube.com/watch?v=17shdT9-wuA
Discovering Neural Nets with Low Kolmogorov Complexity and High Generalization Capability (Neural Networks, 1997)
https://sferics.idsia.ch/pub/juergen/loconet.pdf
The paradox at the heart of mathematics: Gödel's Incompleteness Theorem - Marcus du Sautoy
youtube.com/watch?v=I4pQbo5MQOs
The Philosophy of Science - Hilary Putnam & Bryan Magee (1977)
youtube.com/watch?v=JJB2q8ufAgk
Optimal Ordered Problem Solver
arxiv.org/abs/cs/0207097
Levin's Universal Search from 1973
rjlipton.com/2011/03/14/levins-great-discoveries
https://people.idsia.ch/~juergen/optimalsearch.html
On Learning to Think
arxiv.org/abs/1511.09249
Mindstorms in Natural Language-Based Societies of Mind
Untersuchungen zu dynamischen neuronalen Netzen
https://www.bioinf.jku.at/publications/older/3804.pdf
Evolutionary Principles in Self-Referential Learning
https://people.idsia.ch/~juergen/diploma1987ocr.pdf
Hans-Joachim Bremermann
en.wikipedia.org/wiki/Bremermann%27s_limit
Highway Networks
arxiv.org/abs/1505.00387
https://people.idsia.ch/~juergen/highway-networks.html
The principles of Deep Learning Theory
amzn.to/3WJtPaj
Understanding Deep Learning amzn.to/4doDk63
Discovering Problem Solutions with Low Kolmogorov Complexity and High Generalization Capability (ICML 1995)
https://sferics.idsia.ch/pub/juergen/icmlkolmogorov.pdf
"History of Modern AI and Deep Learning":
https://people.idsia.ch/~juergen/deep-learning-history.html
Jürgen Schmidhuber, the father of generative AI shares his groundbreaking work in deep learning and artificial intelligence. In this exclusive interview, he discusses the history of AI, some of his contributions to the field, and his vision for the future of intelligent machines. Schmidhuber offers unique insights into the exponential growth of technology and the potential impact of AI on humanity and the universe.
MLST is sponsored by Brave:
The Brave Search API covers over 20 billion webpages, built from scratch without Big Tech biases or the recent extortionate price hikes on search API access. Perfect for AI model training and retrieval augmentated generation. Try it now - get 2,000 free queries monthly at brave.com/api.
TOC
00:00:00 Intro
00:03:38 Reasoning
00:13:09 Potential AI Breakthroughs Reducing Computation Needs
00:20:39 Memorization vs. Generalization in AI
00:25:19 Approach to the ARC Challenge
00:29:10 Perceptions of Chat GPT and AGI
00:58:45 Abstract Principles of Jurgen's Approach
01:04:17 Analogical Reasoning and Compression
01:05:48 Breakthroughs in 1991: the P, the G, and the T in ChatGPT and Generative AI
01:15:50 Use of LSTM in Language Models by Tech Giants
01:21:08 Neural Network Aspect Ratio Theory
01:26:53 Reinforcement Learning Without Explicit Teachers
Refs:
★ "Annotated History of Modern AI and Deep Learning" (2022 survey by Schmidhuber):
★ Chain Rule For Backward Credit Assignment (Leibniz, 1676)
★ First Neural Net / Linear Regression / Shallow Learning (Gauss & Legendre, circa 1800)
★ First 20th Century Pioneer of Practical AI (Quevedo, 1914)
★ First Recurrent NN (RNN) Architecture (Lenz, Ising, 1920-1925)
★ AI Theory: Fundamental Limitations of Computation and Computation-Based AI (Gödel, 1931-34)
★ Unpublished ideas about evolving RNNs (Turing, 1948)
★ Multilayer Feedforward NN Without Deep Learning (Rosenblatt, 1958)
★ First Published Learning RNNs (Amari and others, ~1972)
★ First Deep Learning (Ivakhnenko & Lapa, 1965)
★ Deep Learning by Stochastic Gradient Descent (Amari, 1967-68)
★ ReLUs (Fukushima, 1969)
★ Backpropagation (Linnainmaa, 1970); precursor (Kelley, 1960)
★ Backpropagation for NNs (Werbos, 1982)
★ First Deep Convolutional NN (Fukushima, 1979); later combined with Backprop (Waibel 1987, Zhang 1988).
★ Metalearning or Learning to Learn (Schmidhuber, 1987)
★ Generative Adversarial Networks / Artificial Curiosity / NN Online Planners (Schmidhuber, Feb 1990; see the G in Generative AI and ChatGPT)
★ NNs Learn to Generate Subgoals and Work on Command (Schmidhuber, April 1990)
★ NNs Learn to Program NNs: Unnormalized Linear Transformer (Schmidhuber, March 1991; see the T in ChatGPT)
★ Deep Learning by Self-Supervised Pre-Training. Distilling NNs (Schmidhuber, April 1991; see the P in ChatGPT)
★ Experiments with Pre-Training; Analysis of Vanishing/Exploding Gradients, Roots of Long Short-Term Memory / Highway Nets / ResNets (Hochreiter, June 1991, further developed 1999-2015 with other students of Schmidhuber)
★ LSTM journal paper (1997, most cited AI paper of the 20th century)
★ xLSTM (Hochreiter, 2024)
★ Reinforcement Learning Prompt Engineer for Abstract Reasoning and Planning (Schmidhuber 2015)
★ Mindstorms in Natural Language-Based Societies of Mind (2023 paper by Schmidhuber's team)
arxiv.org/abs/2305.17066
★ Bremermann's physical limit of computation (1982)
EXTERNAL LINKS
CogX 2018 - Professor Juergen Schmidhuber
youtube.com/watch?v=17shdT9-wuA
Discovering Neural Nets with Low Kolmogorov Complexity and High Generalization Capability (Neural Networks, 1997)
https://sferics.idsia.ch/pub/juergen/loconet.pdf
The paradox at the heart of mathematics: Gödel's Incompleteness Theorem - Marcus du Sautoy
youtube.com/watch?v=I4pQbo5MQOs
The Philosophy of Science - Hilary Putnam & Bryan Magee (1977)
youtube.com/watch?v=JJB2q8ufAgk
Optimal Ordered Problem Solver
arxiv.org/abs/cs/0207097
Levin's Universal Search from 1973
rjlipton.com/2011/03/14/levins-great-discoveries
https://people.idsia.ch/~juergen/optimalsearch.html
On Learning to Think
arxiv.org/abs/1511.09249
Mindstorms in Natural Language-Based Societies of Mind
Untersuchungen zu dynamischen neuronalen Netzen
https://www.bioinf.jku.at/publications/older/3804.pdf
Evolutionary Principles in Self-Referential Learning
https://people.idsia.ch/~juergen/diploma1987ocr.pdf
Hans-Joachim Bremermann
en.wikipedia.org/wiki/Bremermann%27s_limit
Highway Networks
arxiv.org/abs/1505.00387
https://people.idsia.ch/~juergen/highway-networks.html
The principles of Deep Learning Theory
amzn.to/3WJtPaj
Understanding Deep Learning amzn.to/4doDk63
Discovering Problem Solutions with Low Kolmogorov Complexity and High Generalization Capability (ICML 1995)
https://sferics.idsia.ch/pub/juergen/icmlkolmogorov.pdf
"History of Modern AI and Deep Learning":
https://people.idsia.ch/~juergen/deep-learning-history.html
![Manhattan Project for AI Safety [Connor Leahy]](https://i.ytimg.com/vi/Dt1ySXYTGuA/mqdefault.jpg "Manhattan Project for AI Safety [Connor Leahy]
Connor Leahy and Gabriel Alfour, AI researchers from Conjecture and authors of The Compendium, join us for a critical discussion centered on Artificial Superintelligence (ASI) safety and governance. Drawing from their comprehensive analysis in The Compendium, they articulate a stark warning about the existential risks inherent in uncontrolled AI development, framing it through the lens of intelligence domination—where a sufficiently advanced AI could subordinate humanity, much like humans dominate less intelligent species.
SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.
Goto https://tufalabs.ai/
***
TRANSCRIPT + REFS + NOTES:
https://www.dropbox.com/scl/fi/p86l75y4o2ii40df5t7no/Compendium.pdf?rlkey=tukczgf3flw133sr9rgss0pnj&dl=0
Transcript (new system): https://app.rescript.info/public/share/FObT5GvZYfnixs-aSAD8px4Je_4_aC1kHYG76bvBBRU
https://www.thecompendium.ai/
https://en.wikipedia.org/wiki/Connor_Leahy
https://www.conjecture.dev/about
https://substack.com/@gabecc
TOC:
1. AI Intelligence and Safety Fundamentals
[00:00:00] 1.1 Understanding Intelligence and AI Capabilities
[00:06:20] 1.2 Emergence of Intelligence and Regulatory Challenges
[00:10:18] 1.3 Human vs Animal Intelligence Debate
[00:18:00] 1.4 AI Regulation and Risk Assessment Approaches
[00:26:14] 1.5 Competing AI Development Ideologies
2. Economic and Social Impact
[00:29:10] 2.1 Labor Market Disruption and Post-Scarcity Scenarios
[00:32:40] 2.2 Institutional Frameworks and Tech Power Dynamics
[00:37:40] 2.3 Ethical Frameworks and AI Governance Debates
[00:40:52] 2.4 AI Alignment Evolution and Technical Challenges
3. Technical Governance Framework
[00:55:07] 3.1 Three Levels of AI Safety: Alignment, Corrigibility, and Boundedness
[00:55:30] 3.2 Challenges of AI System Corrigibility and Constitutional Models
[00:57:35] 3.3 Limitations of Current Boundedness Approaches
[00:59:11] 3.4 Abstract Governance Concepts and Policy Solutions
4. Democratic Implementation and Coordination
[00:59:20] 4.1 Governance Design and Measurement Challenges
[01:00:10] 4.2 Democratic Institutions and Experimental Governance
[01:14:10] 4.3 Political Engagement and AI Safety Advocacy
[01:25:30] 4.4 Practical AI Safety Measures and International Coordination
CORE REFS:
[00:01:45] The Compendium (2023), Leahy et al.
https://pdf.thecompendium.ai/the_compendium.pdf
[00:06:50] Geoffrey Hinton Leaves Google, BBC News
https://www.bbc.com/news/world-us-canada-65452940
[00:10:00] ARC-AGI, Chollet
https://arcprize.org/arc-agi
[00:13:25] A Brief History of Intelligence, Bennett
https://www.amazon.com/Brief-History-Intelligence-Humans-Breakthroughs/dp/0063286343
[00:25:35] Statement on AI Risk, Center for AI Safety
https://www.safe.ai/work/statement-on-ai-risk
[00:26:15] Machines of Love and Grace, Amodei
https://darioamodei.com/machines-of-loving-grace
[00:26:35] The Techno-Optimist Manifesto, Andreessen
https://a16z.com/the-techno-optimist-manifesto/
[00:31:55] Techno-Feudalism, Varoufakis
https://www.amazon.co.uk/Technofeudalism-Killed-Capitalism-Yanis-Varoufakis/dp/1847927270
[00:42:40] Introducing Superalignment, OpenAI
https://openai.com/index/introducing-superalignment/
[00:47:20] Three Laws of Robotics, Asimov
https://www.britannica.com/topic/Three-Laws-of-Robotics
[00:50:00] Symbolic AI (GOFAI), Haugeland
https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence
[00:52:30] Intent Alignment, Christiano
https://www.alignmentforum.org/posts/HEZgGBZTpT4Bov7nH/mapping-the-conceptual-territory-in-ai-existential-safety
[00:55:10] Large Language Model Alignment: A Survey, Jiang et al.
http://arxiv.org/pdf/2309.15025
[00:55:40] Constitutional Checks and Balances, Bok
https://plato.stanford.edu/entries/montesquieu/
[00:59:25] Goodharts Law, Manheim & Garrabrant
https://arxiv.org/pdf/1803.04585
[01:00:25] French Constitution Article 37-1, French National Assembly
https://www2.assemblee-nationale.fr/langues/welcome-to-the-english-website-of-the-french-national-assembly
[01:03:00] Intelligence Enhancement and AI Alignment, Yudkowsky
https://intelligence.org/files/AIPosNegFactor.pdf
[01:14:10] UK Government AI Regulation Response, UK Government
https://www.gov.uk/government/consultations/ai-regulation-a-pro-innovation-approach-policy-proposals/outcome/a-pro-innovation-approach-to-ai-regulation-government-response
[01:20:20] French AI Action Summit, Caroli
https://www.csis.org/analysis/frances-ai-action-summit
[01:20:45] Holden Karnofsky Joins Anthropic, Fortune Magazine
https://fortune.com/2025/02/13/anthropic-hired-president-daniela-amodei-husband-ai-safety-responsible-scaling/
[01:25:35] Kill Switch Mechanisms, IEEE/ISO
https://arxiv.org/html/2410.22151v1")
![He Co-Invented the Transformer. Now: Continuous Thought Machines [Llion Jones / Luke Darlow]](https://i.ytimg.com/vi/DtePicx_kFY/mqdefault.jpg "He Co-Invented the Transformer. Now: Continuous Thought Machines [Llion Jones / Luke Darlow]
The Transformer architecture (which powers ChatGPT and nearly all modern AI) might be trapping the industry in a localized rut, preventing us from finding true intelligent reasoning, according to the person who co-invented it. Llion Jones and Luke Darlow, key figures at the research lab Sakana AI, join the show to make this provocative argument, and also introduce new research (CTM) which might lead the way forwards.
We speak about Inventors Remorse & The Trap of Success Despite being one of the original authors of the famous Attention Is All You Need paper that gave birth to the Transformer, Llion explains why he has largely stopped working on them. He argues that the industry is suffering from success capture—because Transformers work so well, everyone is focused on making small tweaks to the same architecture rather than discovering the next big leap.
**SPONSOR MESSAGES START**
—
Build your ideas with AI Studio from Google - http://ai.studio/build
—
Tufa AI Labs is hiring ML Research Engineers https://tufalabs.ai/
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
**END**
The Spiral Problem – Llion uses a striking visual analogy to explain what current AI is missing. If you ask a standard neural network to understand a spiral shape, it solves it by drawing tiny straight lines that just happen to look like a spiral. It fakes the shape without understanding the concept of spiraling. They argue that todays AI models are similar—they are incredible at mimicking intelligent answers without having an internal process of thinking.
Introducing the Continuous Thought Machine (CTM) Luke Darlow deep dives into their solution: a biology-inspired model that fundamentally changes how AI processes information.
The Maze Analogy: Luke explains that standard AI tries to solve a maze by staring at the whole image and guessing the entire path instantly. Their new machine walks through the maze step-by-step.
Thinking Time: This allows the AI to ponder. If a problem is hard, the model can naturally spend more time thinking about it before answering, effectively allowing it to correct its own mistakes and backtrack—something current Language Models struggle to do genuinely.
The pair discuss the culture of Sakana AI, which is modeled after the early days of Google Brain/DeepMind. Llion nostalgically recalls that the Transformer wasnt born from a corporate mandate, but from random people talking over lunch about interesting problems.
https://sakana.ai/
https://x.com/YesThisIsLion
https://x.com/LearningLukeD
TRANSCRIPT:
https://app.rescript.info/public/share/crjzQ-Jo2FQsJc97xsBdfzfOIeMONpg0TFBuCgV2Fu8
TOC:
00:00:00 - Stepping Back from Transformers
00:00:43 - Introduction to Continuous Thought Machines (CTM)
00:01:09 - The Changing Atmosphere of AI Research
00:04:13 - Sakana’s Philosophy: Research Freedom
00:07:45 - The Local Minimum of Large Language Models
00:18:30 - Representation Problems: The Spiral Example
00:29:12 - Technical Deep Dive: CTM Architecture
00:36:00 - Adaptive Computation & Maze Solving
00:47:15 - Model Calibration & Uncertainty
01:00:43 - Sudoku Bench: Measuring True Reasoning
REFS:
Why Greatness Cannot be planned [Kenneth Stanley]
https://www.amazon.co.uk/Why-Greatness-Cannot-Planned-Objective/dp/3319155237
https://www.youtube.com/watch?v=lhYGXYeMq_E
The Hardware Lottery [Sara Hooker]
https://arxiv.org/abs/2009.06489
https://www.youtube.com/watch?v=sQFxbQ7ade0
Continuous Thought Machines [Luke Darlow et al / Sakana]
https://arxiv.org/abs/2505.05522
https://sakana.ai/ctm/
https://youtu.be/5X9cjGLggv0 great walkthrough of algo by Yacine Mahdid
LSTM: The Comeback Story? [Prof. Sepp Hochreiter]
https://www.youtube.com/watch?v=8u2pW2zZLCs
Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis [Kumar/Stanley]
https://arxiv.org/pdf/2505.11581
Intelligent Matrix Exponentiation [Thomas Fischbacher] (Spiral reference)
https://arxiv.org/abs/2008.03936
A Spline Theory of Deep Networks [Randall Balestriero]
https://proceedings.mlr.press/v80/balestriero18b/balestriero18b.pdf
https://www.youtube.com/watch?v=86ib0sfdFtw
https://www.youtube.com/watch?v=l3O2J3LMxqI
On the Biology of a Large Language Model [Anthropic, Jack Lindsey et al]
https://transformer-circuits.pub/2025/attribution-graphs/biology.html
The ARC Prize 2024 Winning Algorithm [Daniel Franzen and Jan Disselhoff] “The ARChitects”
https://www.youtube.com/watch?v=mTX_sAq zY
Neural Turing Machine [Graves]
https://arxiv.org/pdf/1410.5401
Adaptive Computation Time for Recurrent Neural Networks [Graves]
https://arxiv.org/abs/1603.08983
Sudoko Bench [Sakana]
https://pub.sakana.ai/sudoku/")
![Transformers Need Glasses! [Federico Barbero]](https://i.ytimg.com/vi/FAspMnu4Rt0/mqdefault.jpg "Transformers Need Glasses! [Federico Barbero]
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!
https://centml.ai/pricing/
Federico Barbero (DeepMind/Oxford) is the lead author of Transformers Need Glasses!, a paper revealing fundamental architectural limitations in transformer-based language models. The conversation explores why LLMs fail at seemingly trivial tasks like copying the last token of a sequence or counting repeated elements, tracing these failures to representation collapse — where internal representations of distinct inputs converge below machine precision as context length grows. Federico connects these findings to information propagation theory from graph neural networks, showing how causal attention creates an inherent bias toward the start of a sequence, while training dynamics push models to attend to the most recent tokens, leaving the middle of long contexts as a blind spot. The discussion covers connections to spectral graph theory, heat equations on graphs, Petar Veličkovićs work on graph attention networks, the role of softmax in limiting sharp attention, and practical glasses — architectural tweaks that can help transformers see more clearly.
REFERENCES:
paper:
[00:01:05] Transformers Need Glasses!
https://proceedings.neurips.cc/paper_files/paper/2024/file/b1d35561c4a4a0e0b6012b2af531e149-Paper-Conference.pdf
[00:05:30] Softmax is Not Enough
https://arxiv.org/abs/2410.01104
[00:15:05] Graph Attention Networks
https://arxiv.org/abs/1710.10903
[00:43:23] Neural Networks and the Chomsky Hierarchy
https://arxiv.org/abs/2207.02098
[00:51:04] On the Measure of Intelligence
https://arxiv.org/abs/1911.01547
[01:00:35] Epistemic Foraging
https://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2016.00056/full
LINKS:
Full Transcript: https://app.rescript.info/share/d43b3918dc7cb8c8b822ebef40b8f66f
Download PDF transcript: https://app.rescript.info/api/public/sessions/53f7d10ff5c72eca/pdf")
![MIGHT THE ROBOTS TAKE OVER? [Prof. Yoshua Bengio]](https://i.ytimg.com/vi/G1ARvwQntAU/mqdefault.jpg "MIGHT THE ROBOTS TAKE OVER? [Prof. Yoshua Bengio]
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.
https://centml.ai/pricing/
Turing Award winner Professor Yoshua Bengio sits down with Tim Scarfe for a wide-ranging discussion on AI safety, the architecture of intelligence, and what happens when machines become as capable as our best researchers.
Bengio argues that the current race toward agentic AI systems carries fundamental risks that most people underestimate. He lays out how instrumental convergence and reward tampering could lead systems to develop goals misaligned with human interests not through malice, but through the basic logic of optimization. His proposed alternative: powerful AI tools that function as scientific oracles rather than autonomous agents, systems that can revolutionize medicine and science without needing goals of their own.
The conversation covers international AI governance and the game theory driving the US-China competition, the military implications of frontier AI capabilities, and why hardware-enabled governance may be the most promising path toward verifiable AI treaties. Bengio draws direct parallels to nuclear nonproliferation and warns that data centers will become strategic military assets.
On the technical side, Bengio discusses the return of recurrent architectures (the Were RNNs All We Needed? paper), his work on GFlowNets for probabilistic inference with discrete structures, a complexity-based theory of compositionality that tries to formalize what was previously just intuition, and the question of whether System 2 reasoning needs to be designed from first principles rather than bolted onto existing architectures.
Throughout, Bengio repeatedly emphasizes what he does not know a deliberate stance he contrasts with the overconfidence he sees driving dangerous policy decisions.
REFERENCES:
paper:
[00:00:15] AI Risk Statement
https://www.safe.ai/work/statement-on-ai-risk
[00:23:10] Reward Tampering and AI Safety
https://hdsr.mitpress.mit.edu/pub/w974bwb0
[00:44:30] Can a Bayesian Oracle Prevent Harm?
https://arxiv.org/abs/2408.05284
[00:52:00] Hardware-Enabled AI Governance Memo
https://yoshuabengio.org/wp-content/uploads/2024/08/FlexHEG-Memo_August-2024.pdf
[01:33:20] GFlowNet Foundations
https://arxiv.org/abs/2111.09266
[01:35:05] Complexity-Based Compositionality Theory
https://arxiv.org/abs/2410.14817
[01:37:50] Discrete Attractor States in Neural Systems
https://arxiv.org/abs/2302.06403
person:
[00:03:50] Professor Yoshua Bengio
https://yoshuabengio.org/
tool:
[00:40:45] Munk Debate on AI Existential Risk
https://munkdebates.com/debates/artificial-intelligence
[00:56:07] 2018 Turing Award
https://awards.acm.org/about/2018-turing
[01:12:35] EU AI Act Code of Practice
https://digital-strategy.ec.europa.eu/en/news/meet-chairs-leading-development-first-general-purpose-ai-code-practice
LINKS:
Full Transcript: https://app.rescript.info/share/1c4f010852d0db9a34dbeb043c0da26b
Download PDF transcript: https://app.rescript.info/api/public/sessions/baff3f8846202250/pdf
Yoshua Bengio:
https://x.com/Yoshua_Bengio
https://scholar.google.com/citations?user=kukA0LcAAAAJ&hl=en
https://yoshuabengio.org/
https://en.wikipedia.org/wiki/Yoshua_Bengio")
