Uploaded November 2024 | Updated September 2026, 1 week ago
Eliezer Yudkowsky and Stephen Wolfram discuss artificial intelligence and its potential existen‑
tial risks. They traversed fundamental questions about AI safety, consciousness, computational irreducibility, and the nature of intelligence.
The discourse centered on Yudkowsky’s argument that advanced AI systems pose an existential threat to humanity, primarily due to the challenge of alignment and the potential for emergent goals that diverge from human values. Wolfram, while acknowledging potential risks, approached the topic from a his signature measured perspective, emphasizing the importance of understanding computational systems’ fundamental nature and questioning whether AI systems would necessarily develop the kind of goal‑directed behavior Yudkowsky fears.
SHOWNOTES (transcription, references, summary, best quotes etc):
dropbox.com/scl/fi/3st8dts2ba7yob161dchd/EliezerWolfram.pdf?rlkey=b6va5j8upgqwl9s2muc924vtt&st=vemwqx7a&dl=0
***
MLST IS SPONSORED BY TUFA AI LABS!
The current winners of the ARC challenge, MindsAI are part of Tufa AI Labs. They are hiring ML engineers. Are you interested?! Please goto tufalabs.ai
***
en.wikipedia.org/wiki/Eliezer_Yudkowsky
en.wikipedia.org/wiki/Stephen_Wolfram
TOC:
1. Foundational AI Concepts and Risks
[00:00:00] 1.1 AI Optimization and System Capabilities Debate
[00:06:46] 1.2 Computational Irreducibility and Intelligence Limitations
[00:20:09] 1.3 Existential Risk and Species Succession
[00:23:28] 1.4 Consciousness and Value Preservation in AI Systems
2. Ethics and Philosophy in AI
[00:33:24] 2.1 Moral Value of Human Consciousness vs. Computation
[00:36:30] 2.2 Ethics and Moral Philosophy Debate
[00:39:58] 2.3 Existential Risks and Digital Immortality
[00:43:30] 2.4 Consciousness and Personal Identity in Brain Emulation
3. Truth and Logic in AI Systems
[00:54:39] 3.1 AI Persuasion Ethics and Truth
[01:01:48] 3.2 Mathematical Truth and Logic in AI Systems
[01:11:29] 3.3 Universal Truth vs Personal Interpretation in Ethics and Mathematics
[01:14:43] 3.4 Quantum Mechanics and Fundamental Reality Debate
4. AI Capabilities and Constraints
[01:21:21] 4.1 AI Perception and Physical Laws
[01:28:33] 4.2 AI Capabilities and Computational Constraints
[01:34:59] 4.3 AI Motivation and Anthropomorphization Debate
[01:38:09] 4.4 Prediction vs Agency in AI Systems
5. AI System Architecture and Behavior
[01:44:47] 5.1 Computational Irreducibility and Probabilistic Prediction
[01:48:10] 5.2 Teleological vs Mechanistic Explanations of AI Behavior
[02:09:41] 5.3 Machine Learning as Assembly of Computational Components
[02:29:52] 5.4 AI Safety and Predictability in Complex Systems
6. Goal Optimization and Alignment
[02:50:30] 6.1 Goal Specification and Optimization Challenges in AI Systems
[02:58:31] 6.2 Intelligence, Computation, and Goal-Directed Behavior
[03:02:18] 6.3 Optimization Goals and Human Existential Risk
[03:08:49] 6.4 Emergent Goals and AI Alignment Challenges
7. AI Evolution and Risk Assessment
[03:19:44] 7.1 Inner Optimization and Mesa-Optimization Theory
[03:34:00] 7.2 Dynamic AI Goals and Extinction Risk Debate
[03:56:05] 7.3 AI Risk and Biological System Analogies
[04:09:37] 7.4 Expert Risk Assessments and Optimism vs Reality
8. Future Implications and Economics
[04:13:01] 8.1 Economic and Proliferation Considerations
Eliezer Yudkowsky and Stephen Wolfram discuss artificial intelligence and its potential existen‑
tial risks. They traversed fundamental questions about AI safety, consciousness, computational irreducibility, and the nature of intelligence.
The discourse centered on Yudkowsky’s argument that advanced AI systems pose an existential threat to humanity, primarily due to the challenge of alignment and the potential for emergent goals that diverge from human values. Wolfram, while acknowledging potential risks, approached the topic from a his signature measured perspective, emphasizing the importance of understanding computational systems’ fundamental nature and questioning whether AI systems would necessarily develop the kind of goal‑directed behavior Yudkowsky fears.
SHOWNOTES (transcription, references, summary, best quotes etc):
dropbox.com/scl/fi/3st8dts2ba7yob161dchd/EliezerWolfram.pdf?rlkey=b6va5j8upgqwl9s2muc924vtt&st=vemwqx7a&dl=0
***
MLST IS SPONSORED BY TUFA AI LABS!
The current winners of the ARC challenge, MindsAI are part of Tufa AI Labs. They are hiring ML engineers. Are you interested?! Please goto tufalabs.ai
***
en.wikipedia.org/wiki/Eliezer_Yudkowsky
en.wikipedia.org/wiki/Stephen_Wolfram
TOC:
1. Foundational AI Concepts and Risks
[00:00:00] 1.1 AI Optimization and System Capabilities Debate
[00:06:46] 1.2 Computational Irreducibility and Intelligence Limitations
[00:20:09] 1.3 Existential Risk and Species Succession
[00:23:28] 1.4 Consciousness and Value Preservation in AI Systems
2. Ethics and Philosophy in AI
[00:33:24] 2.1 Moral Value of Human Consciousness vs. Computation
[00:36:30] 2.2 Ethics and Moral Philosophy Debate
[00:39:58] 2.3 Existential Risks and Digital Immortality
[00:43:30] 2.4 Consciousness and Personal Identity in Brain Emulation
3. Truth and Logic in AI Systems
[00:54:39] 3.1 AI Persuasion Ethics and Truth
[01:01:48] 3.2 Mathematical Truth and Logic in AI Systems
[01:11:29] 3.3 Universal Truth vs Personal Interpretation in Ethics and Mathematics
[01:14:43] 3.4 Quantum Mechanics and Fundamental Reality Debate
4. AI Capabilities and Constraints
[01:21:21] 4.1 AI Perception and Physical Laws
[01:28:33] 4.2 AI Capabilities and Computational Constraints
[01:34:59] 4.3 AI Motivation and Anthropomorphization Debate
[01:38:09] 4.4 Prediction vs Agency in AI Systems
5. AI System Architecture and Behavior
[01:44:47] 5.1 Computational Irreducibility and Probabilistic Prediction
[01:48:10] 5.2 Teleological vs Mechanistic Explanations of AI Behavior
[02:09:41] 5.3 Machine Learning as Assembly of Computational Components
[02:29:52] 5.4 AI Safety and Predictability in Complex Systems
6. Goal Optimization and Alignment
[02:50:30] 6.1 Goal Specification and Optimization Challenges in AI Systems
[02:58:31] 6.2 Intelligence, Computation, and Goal-Directed Behavior
[03:02:18] 6.3 Optimization Goals and Human Existential Risk
[03:08:49] 6.4 Emergent Goals and AI Alignment Challenges
7. AI Evolution and Risk Assessment
[03:19:44] 7.1 Inner Optimization and Mesa-Optimization Theory
[03:34:00] 7.2 Dynamic AI Goals and Extinction Risk Debate
[03:56:05] 7.3 AI Risk and Biological System Analogies
[04:09:37] 7.4 Expert Risk Assessments and Optimism vs Reality
8. Future Implications and Economics
[04:13:01] 8.1 Economic and Proliferation Considerations
![Do you think that ChatGPT can reason? [Prof. Subbarao Kambhampati]](https://i.ytimg.com/vi/y1WnHpedi2A/mqdefault.jpg "Do you think that ChatGPT can reason? [Prof. Subbarao Kambhampati]
Prof. Subbarao Kambhampati argues that while LLMs are impressive and useful tools, especially for creative tasks, they have fundamental limitations in logical reasoning and cannot provide guarantees about the correctness of their outputs. He advocates for hybrid approaches that combine LLMs with external verification systems.
MLST is sponsored by Brave:
The Brave Search API covers over 20 billion webpages, built from scratch without Big Tech biases or the recent extortionate price hikes on search API access. Perfect for AI model training and retrieval augmentated generation. Try it now - get 2,000 free queries monthly at http://brave.com/api.
This is 2/13 of our #ICML2024 series
TOC
[00:00:00] Intro
[00:02:06] Bio
[00:03:02] LLMs are n-gram models on steroids
[00:07:26] Is natural language a formal language?
[00:08:34] Natural language is formal?
[00:11:01] Do LLMs reason?
[00:19:13] Definition of reasoning
[00:31:40] Creativity in reasoning
[00:50:27] Chollets ARC challenge
[01:01:31] Can we reason without verification?
[01:10:00] LLMs cant solve some tasks
[01:19:07] LLM Modulo framework
[01:29:26] Future trends of architecture
[01:34:48] Future research directions
Pod: https://podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/Prof Subbarao-Kambhampati LLMs-dont-reason they-memorize-ICML2024-213-e2mjcse
Subbarao Kambhampati:
https://x.com/rao2z
Interviewer: Dr. Tim Scarfe
Refs:
Can LLMs Really Reason and Plan?
https://cacm.acm.org/blogcacm/can-llms-really-reason-and-plan/
On the Planning Abilities of Large Language Models : A Critical Investigation
https://arxiv.org/pdf/2305.15771
Chain of Thoughtlessness? An Analysis of CoT in Planning
https://arxiv.org/pdf/2405.04776
On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
https://arxiv.org/pdf/2402.08115
LLMs Cant Plan, But Can Help Planning in LLM-Modulo Frameworks
https://arxiv.org/pdf/2402.01817
Embers of Autoregression: Understanding Large Language
Models Through the Problem They are Trained to Solve
https://arxiv.org/pdf/2309.13638
https://arxiv.org/abs/2402.04210
Task Success is not Enough
Faith and Fate: Limits of Transformers on Compositionality finetuning multiplication with four digit numbers (added after pub)
https://arxiv.org/pdf/2305.18654
Partition function (number theory) (Srinivasa Ramanujan and G.H. Hardys work)
https://en.wikipedia.org/wiki/Partition_function_(number_theory)
Poincaré conjecture
https://en.wikipedia.org/wiki/Poincar%C3%A9_conjecture
Gödels incompleteness theorems
https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems
ROT13 (Rotate13, rotate by 13 places)
https://en.wikipedia.org/wiki/ROT13
A Mathematical Theory of Communication (C. E. SHANNON)
https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf
Sparks of AGI
https://arxiv.org/abs/2303.12712
Kambhampati thesis on speech recognition (1983)
https://rakaposhi.eas.asu.edu/rao-btech-thesis.pdf
PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
https://arxiv.org/abs/2206.10498
Explainable human-AI interaction
https://link.springer.com/book/10.1007/978-3-031-03767-2
Tree of Thoughts
https://arxiv.org/abs/2305.10601
On the Measure of Intelligence (ARC Challenge)
https://arxiv.org/abs/1911.01547
Getting 50% (SoTA) on ARC-AGI with GPT-4o (Ryan Greenblatt ARC solution)
https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt
PROGRAMS WITH COMMON SENSE (John McCarthy) - AI should be an advice taker program
https://www.cs.cornell.edu/selman/cs672/readings/mccarthy-upd.pdf
Original chain of thought paper
https://arxiv.org/abs/2201.11903
ICAPS 2024 Keynote: Dale Schuurmans on Computing and Planning with Large Generative Models (COT)
https://www.youtube.com/watch?v=YnMqbpdHcaY
The Hardware Lottery (Hooker)
https://arxiv.org/abs/2009.06489
A Path Towards Autonomous Machine Intelligence (JEPA/LeCun)
https://openreview.net/pdf?id=BZ5a1r-kVsf
AlphaGeometry
https://www.nature.com/articles/s41586-023-06747-5
FunSearch
https://www.nature.com/articles/s41586-023-06924-6
Emergent Abilities of Large Language Models
https://arxiv.org/abs/2206.07682
Language models are not naysayers (Negation in LLMs)
https://arxiv.org/abs/2306.08189
The Reversal Curse: LLMs trained on A is B fail to learn B is A
https://arxiv.org/abs/2309.12288
Embracing negative results
https://openreview.net/forum?id=3RXAiU7sss")
![Moving Beyond Surface Statistics (Apple researcher) [Iman Mirzadeh]](https://i.ytimg.com/vi/yQPduek-Q5s/mqdefault.jpg "Moving Beyond Surface Statistics (Apple researcher) [Iman Mirzadeh]
Iman Mirzadeh is a machine learning research engineer at Apple and the lead author of the GSM-Symbolic paper, which exposed deep fragility in how large language models handle mathematical reasoning. In this conversation, he draws a sharp line between intelligence and achievement between what a system can score on a benchmark and what it actually understands.
The discussion starts with chess. Mirzadeh explains how grandmasters dont use engines to memorize moves; they use them to develop theory. AlphaZero discovered unprecedented strategies, but that knowledge stays trapped in the game. Humans, by contrast, extract abstract principles like control the center and transfer them to entirely different domains. That capacity for abstraction is what he thinks current AI architecturally lacks.
His critique of LLMs is structural. These systems are trained to minimize cross-entropy loss over a distribution, and by construction they cannot reason beyond what that distribution contains. Change the surface form of a problem swap names, add irrelevant clauses and performance varies wildly, even on grade-school math.
Thats the core finding of GSM-Symbolic. By generating templated variants of math word problems, Mirzadehs team showed that even frontier models exhibit large performance variance from changes that should be semantically irrelevant. The implication: what looks like reasoning is closer to sophisticated pattern matching across memorized distributions.
Mirzadeh proposes that intelligence should be measured by the slope of a systems scaling how fast it can learn novel things rather than its current benchmark position. The conversation also covers the connectionism-symbolism divide, active engagement and agency as necessary conditions for learning, and why we might need a fundamentally different vessel to reach genuine reasoning.
SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.
TIMESTAMPS:
00:00:00 Intelligence vs Achievement in AI
00:03:27 AlphaZero and Abstract Understanding in Chess
00:10:10 Language Models as Distribution Learners
00:14:47 The State of AI Research Methodology
00:24:24 Interpolation vs True Reasoning in LLMs
00:29:00 Measuring Intelligence: From Chollet to the Iman Moon Test
00:35:35 Agency, Active Learning, and World Models
00:47:15 Scaling Laws and the Connectionism-Symbolism Debate
00:58:09 GSM-Symbolic: Exposing LLM Reasoning Fragility
REFERENCES:
paper:
[00:00:55] Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
https://arxiv.org/abs/1712.01815
[00:17:15] GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
https://arxiv.org/abs/2410.05229
[00:21:20] Connectionism and Cognitive Architecture: A Critical Analysis
https://www.sciencedirect.com/science/article/pii/001002779090014B
[00:29:35] On the Measure of Intelligence
https://arxiv.org/abs/1911.01547
[00:33:25] On definition of intelligence
https://www.sciencedirect.com/science/article/pii/S0160289624000266
[00:35:25] Defining Intelligence
https://cis.temple.edu/~wangp/papers.html
[00:43:10] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
https://arxiv.org/abs/2201.11903
[00:47:45] Scaling Laws for Neural Language Models
https://arxiv.org/abs/2001.08361
[00:55:10] Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems
https://www.sciencedirect.com/science/article/abs/pii/000437029090007M
book:
[00:07:05] Game Changer: AlphaZeros Groundbreaking Chess Strategies
https://www.amazon.com/Game-Changer-AlphaZeros-Groundbreaking-Strategies/dp/9056918184
[00:37:35] How We Learn: Why Brains Learn Better Than Any Machine... for Now
https://www.amazon.com/How-We-Learn-Brains-Machine/dp/0525559884
[00:39:30] Surfaces and Essences: Analogy as the Fuel and Fire of Thinking
https://www.amazon.com/Surfaces-Essences-Analogy-Fuel-Thinking/dp/0465018475
reference:
[00:11:30] NLP Course: Language Modeling
http://lena-voita.github.io/nlp_course/language_modeling.html
[01:08:40] GSM8K: Training Verifiers to Solve Math Word Problems
https://huggingface.co/datasets/openai/gsm8k
LINKS:
Full Transcript: https://app.rescript.info/share/72689965572c7fd460954f70e26b2eaa
Download PDF transcript: https://app.rescript.info/api/public/sessions/abf4f57b59fcbbd5/pdf")
