Uploaded September 2024 | Updated September 2026, 1 week ago
MLST is sponsored by Brave:
The Brave Search API covers over 20 billion webpages, built from scratch without Big Tech biases or the recent extortionate price hikes on search API access. Perfect for AI model training and retrieval augmentated generation. Try it now - get 2,000 free queries monthly at brave.com/api.
David Hanson, CEO of Hanson Robotics and creator of the humanoid robot Sophia, sits down with Tim Scarfe at AGI-24 to talk about building AI systems that don't just process information but actually care about the world around them.
Hanson's background is unusual for a roboticist. He trained across neuroscience, AI, the arts, sculpture, and material science -- all of which fed into making Sophia's facial expressions more lifelike than anything else on the market. His PhD combined aesthetics, cognitive science, and mechanical engineering. The result: robots that people instinctively want to talk to.
The conversation gets philosophically dense quickly. Hanson argues that current LLMs, while useful as "cortical prostheses" that democratize expertise, fundamentally lack what makes biological intelligence work: drives. Not high-level goal-setting, but the deep, low-level motivation to exist and persist that cells have had since life began. He calls this "appreciation" -- the thing that makes an organism actually struggle to survive rather than sit dormant.
This leads to his central thesis: if you want AI with genuine agency, you need to wire in drives analogous to biological ones, not at the high level where they become brittle, but at a low, flexible level where emergence can do the heavy lifting. He's not talking about copying brains. He's talking about identifying the core evolutionary principles -- the desire to live, curiosity, the appreciation of patterns -- and encoding them in computational systems.
The ethical implications occupy much of the back half. Hanson introduces what he calls "existential pattern ethics" -- the idea that moral behavior might be grounded in the fundamental bioinformatics of physics and math, in the tendency of certain patterns to come into existence and persist. This isn't standard AI safety talk. He's proposing that ethics isn't just about restrictions ("don't do this") but about growth, play, and creative exploration. His goal isn't artificial super intelligence alone -- it's "super wisdom."
The interview touches on human-AI integration, the risks of technological augmentation widening inequality, democratizing AI globally, and the potential for AI-enhanced mental health tools. Hanson filmed this at AGI-24 and throughout, he walks a line between visionary and pragmatic, acknowledging we're still in the "tinkering" phase while making the case that the tinkering needs to be guided by something deeper than performance benchmarks.
---
TIMESTAMPS:
00:00:00 Introduction and Sizzle Reel
00:01:48 David Hanson's Interdisciplinary Background
00:03:27 Sophia and Human-Robot Social Interaction
00:05:55 Compassion as the Distinguishing Factor
00:09:54 AI as Cortical Prosthesis
00:13:17 Biological Drives as the Foundation for AGI
00:20:34 Creating AI with Genuine Agency
00:23:23 Flexible Low-Level Desires vs Brittle High-Level Goals
00:27:53 Enhancing Humanity Through AI
00:30:14 Existential Pattern Ethics
00:35:35 Morality Beyond Restrictions
00:38:07 Democratizing AI Technologies Globally
00:43:37 Human-AI Integration and Identity
00:50:03 Technological Augmentation, Inequality, and Corporate Ethics
---
REFERENCES:
reference:
[00:00:00] AGI-24 Conference
agi-conference.org
[00:01:48] Bina48 Robot
en.wikipedia.org/wiki/Bina48
[00:03:27] Sophia the Robot
youtube.com/watch?v=9u1O954cMmE
[00:03:27] Eusociality in Human Cognition
en.wikipedia.org/wiki/Eusociality
[00:43:37] Integrated Information Theory (IIT)
en.wikipedia.org/wiki/Integrated_information_theory
person:
[00:01:48] David Hanson - Hanson Robotics
hansonrobotics.com/david-hanson
[00:43:37] Susan Schneider - Artificial You
en.wikipedia.org/wiki/Susan_Schneider
book:
[00:05:55] Philip K. Dick - Do Androids Dream of Electric Sheep?
en.wikipedia.org/wiki/Do_Androids_Dream_of_Electric_Sheep%3F
---
LINKS:
Full Transcript: app.rescript.info/share/16daa6558770b8fbf3cd9b871a181ba2
Download PDF transcript: app.rescript.info/api/public/sessions/ec445a79d5f896dd/pdf
MLST is sponsored by Brave:
The Brave Search API covers over 20 billion webpages, built from scratch without Big Tech biases or the recent extortionate price hikes on search API access. Perfect for AI model training and retrieval augmentated generation. Try it now - get 2,000 free queries monthly at brave.com/api.
David Hanson, CEO of Hanson Robotics and creator of the humanoid robot Sophia, sits down with Tim Scarfe at AGI-24 to talk about building AI systems that don't just process information but actually care about the world around them.
Hanson's background is unusual for a roboticist. He trained across neuroscience, AI, the arts, sculpture, and material science -- all of which fed into making Sophia's facial expressions more lifelike than anything else on the market. His PhD combined aesthetics, cognitive science, and mechanical engineering. The result: robots that people instinctively want to talk to.
The conversation gets philosophically dense quickly. Hanson argues that current LLMs, while useful as "cortical prostheses" that democratize expertise, fundamentally lack what makes biological intelligence work: drives. Not high-level goal-setting, but the deep, low-level motivation to exist and persist that cells have had since life began. He calls this "appreciation" -- the thing that makes an organism actually struggle to survive rather than sit dormant.
This leads to his central thesis: if you want AI with genuine agency, you need to wire in drives analogous to biological ones, not at the high level where they become brittle, but at a low, flexible level where emergence can do the heavy lifting. He's not talking about copying brains. He's talking about identifying the core evolutionary principles -- the desire to live, curiosity, the appreciation of patterns -- and encoding them in computational systems.
The ethical implications occupy much of the back half. Hanson introduces what he calls "existential pattern ethics" -- the idea that moral behavior might be grounded in the fundamental bioinformatics of physics and math, in the tendency of certain patterns to come into existence and persist. This isn't standard AI safety talk. He's proposing that ethics isn't just about restrictions ("don't do this") but about growth, play, and creative exploration. His goal isn't artificial super intelligence alone -- it's "super wisdom."
The interview touches on human-AI integration, the risks of technological augmentation widening inequality, democratizing AI globally, and the potential for AI-enhanced mental health tools. Hanson filmed this at AGI-24 and throughout, he walks a line between visionary and pragmatic, acknowledging we're still in the "tinkering" phase while making the case that the tinkering needs to be guided by something deeper than performance benchmarks.
---
TIMESTAMPS:
00:00:00 Introduction and Sizzle Reel
00:01:48 David Hanson's Interdisciplinary Background
00:03:27 Sophia and Human-Robot Social Interaction
00:05:55 Compassion as the Distinguishing Factor
00:09:54 AI as Cortical Prosthesis
00:13:17 Biological Drives as the Foundation for AGI
00:20:34 Creating AI with Genuine Agency
00:23:23 Flexible Low-Level Desires vs Brittle High-Level Goals
00:27:53 Enhancing Humanity Through AI
00:30:14 Existential Pattern Ethics
00:35:35 Morality Beyond Restrictions
00:38:07 Democratizing AI Technologies Globally
00:43:37 Human-AI Integration and Identity
00:50:03 Technological Augmentation, Inequality, and Corporate Ethics
---
REFERENCES:
reference:
[00:00:00] AGI-24 Conference
agi-conference.org
[00:01:48] Bina48 Robot
en.wikipedia.org/wiki/Bina48
[00:03:27] Sophia the Robot
youtube.com/watch?v=9u1O954cMmE
[00:03:27] Eusociality in Human Cognition
en.wikipedia.org/wiki/Eusociality
[00:43:37] Integrated Information Theory (IIT)
en.wikipedia.org/wiki/Integrated_information_theory
person:
[00:01:48] David Hanson - Hanson Robotics
hansonrobotics.com/david-hanson
[00:43:37] Susan Schneider - Artificial You
en.wikipedia.org/wiki/Susan_Schneider
book:
[00:05:55] Philip K. Dick - Do Androids Dream of Electric Sheep?
en.wikipedia.org/wiki/Do_Androids_Dream_of_Electric_Sheep%3F
---
LINKS:
Full Transcript: app.rescript.info/share/16daa6558770b8fbf3cd9b871a181ba2
Download PDF transcript: app.rescript.info/api/public/sessions/ec445a79d5f896dd/pdf
![The Real Reason Huge AI Models Actually Work [Prof. Andrew Wilson]](https://i.ytimg.com/vi/M-jTeBCEGHc/mqdefault.jpg "The Real Reason Huge AI Models Actually Work [Prof. Andrew Wilson]
Why can billion-parameter models perform so well without catastrophically overfitting? The answer lies in the mysterious simplicity bias that emerges at scale, a core concept of the double descent phenomenon.
Professor Andrew Wilson from NYU explains why many common-sense ideas in artificial intelligence might be wrong. For decades, the rule of thumb in machine learning has been to fear complexity. The thinking goes: if your model has too many parameters (is too complex) for the amount of data you have, it will overfit by essentially memorizing the data instead of learning the underlying patterns. This leads to poor performance on new, unseen data. This is known as the classic bias-variance trade-off i.e. a balancing act between a model thats too simple and one thats too complex.
**SPONSOR MESSAGES**
—
Tufa AI Labs is an AI research lab based in Zurich. **They are hiring ML research engineers!**
This is a once in a lifetime opportunity to work with one of the best labs in Europe
Contact Benjamin Crouzier - https://tufalabs.ai/
—
Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
Description Continued:
Professor Wilson challenges this fundamental belief (fearing complexity). He makes a few surprising points:
**Bigger Can Be Better**: massive models dont just get more flexible; they also develop a stronger simplicity bias. So, if your model is overfitting, the solution might paradoxically be to make it even bigger.
**The Bias-Variance Trade-off is a Misnomer**: Wilson claims you dont actually have to trade one for the other. You can have a model that is incredibly expressive and flexible while also being strongly biased toward simple solutions. He points to the double descent phenomenon, where performance first gets worse as models get more complex, but then surprisingly starts getting better again.
**Honest Beliefs and Bayesian Thinking**: His core philosophy is that we should build models that honestly represent our beliefs about the world. We believe the world is complex, so our models should be expressive. But we also believe in Occams razor—that the simplest explanation is often the best. He champions Bayesian methods, which naturally balance these two ideas through a process called marginalization, which he describes as an automatic Occams razor.
TOC:
[00:00:00] Introduction and Thesis
[00:04:19] Challenging Conventional Wisdom
[00:11:17] The Philosophy of a Scientist-Engineer
[00:16:47] Expressiveness, Overfitting, and Bias
[00:28:15] Understanding, Compression, and Kolmogorov Complexity
[01:05:06] The Surprising Power of Generalization
[01:13:21] The Elegance of Bayesian Inference
[01:33:02] The Geometry of Learning
[01:46:28] Practical Advice and The Future of AI
Prof. Andrew Gordon Wilson:
https://x.com/andrewgwils
https://cims.nyu.edu/~andrewgw/
https://scholar.google.com/citations?user=twWX2LIAAAAJ&hl=en
https://www.youtube.com/watch?v=Aja0kZeWRy4
https://www.youtube.com/watch?v=HEp4TOrkwV4
TRANSCRIPT:
https://app.rescript.info/public/share/H4Io1Y7Rr54MM05FuZgAv4yphoukCfkqokyzSYJwCK8
REFS:
Deep Learning is Not So Mysterious or Different [Andrew Gordon Wilson]
https://arxiv.org/abs/2503.02113
Bayesian Deep Learning and a Probabilistic Perspective of Generalization [Andrew Gordon Wilson, Pavel Izmailov]
https://arxiv.org/abs/2002.08791
Compute-Optimal LLMs Provably Generalize Better With Scale [Marc Finzi, Sanyam Kapoor, Diego Granziol, Anming Gu, Christopher De Sa, J. Zico Kolter, Andrew Gordon Wilson]
https://arxiv.org/abs/2504.15208")
![ARC Prize Version 2 Launch Video! [Francois Chollet, Mike Knoop]](https://i.ytimg.com/vi/M3b59lZYBW8/mqdefault.jpg "ARC Prize Version 2 Launch Video! [Francois Chollet, Mike Knoop]
Francois Chollet and Mike Knoop join Tim Scarfe to announce ARC-AGI 2 and the ARC Prize 2025 contest. The new benchmark has been human-calibrated with 400 participants and adversarially designed so that frontier reasoning models score in the single digits, while every task remains solvable by at least two humans. The conversation covers the story behind testing OpenAI o3 on ARC v1 (where it scored 75-85%), what those results reveal about fluid intelligence versus brute-force capability, and why Chollet considers o3 the first model with genuine if limited fluid intelligence. Technical discussion ranges from chain-of-thought reasoning as natural language program synthesis, to the exponential failure modes of current reasoning systems on spatially complex tasks, to the deeper question of whether intelligence is best understood as a binary category or a spectrum of recombination depth. Chollet argues that intelligence is fundamentally about efficiency of knowledge acquisition and adaptation, not raw capability, and that ARC-AGI 2 is designed to measure exactly this distinction.
SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.
REFERENCES:
Paper:
[00:00:15] On the Measure of Intelligence
https://arxiv.org/abs/1911.01547
[00:12:50] OpenAI o3 Performance on ARC v1
https://arcprize.org/blog/oai-o3-pub-breakthrough
[00:18:30] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
https://arxiv.org/abs/2201.11903
[00:26:05] ARC Prize 2024: Technical Report
https://arxiv.org/abs/2412.04604
[00:48:57] The Bitter Lesson
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
[00:53:30] Decoding Strategies in Neural Text Generation
https://www.mdpi.com/2078-2489/12/9/355/pdf
Organization:
[00:06:45] ARC Prize Foundation
https://arcprize.org/
LINKS:
Full Transcript: https://app.rescript.info/share/19ed0e8921636ce804d8780e6b03f67d
Download PDF transcript: https://app.rescript.info/api/public/sessions/302bc045019e056a/pdf")
![You dont fine-tune your way to AGI - Heres why. [Eiso Kant]](https://i.ytimg.com/vi/NDrosuKhXeo/mqdefault.jpg "You dont fine-tune your way to AGI - Heres why. [Eiso Kant]
Eiso Kant is the co-founder and CTO of Poolside AI, one of roughly seven companies worldwide with the technical muscle to build frontier foundation models from scratch. He sat down with Tim to explain why Poolside deliberately rejected the prevailing wisdom of just scale up the next GPT and instead bet the company on a thesis most labs were ignoring: that reinforcement learning from code execution feedback is the missing scaling axis.
The argument is straightforward and, once you hear it, hard to unsee. Next-token prediction is imitation learning. Reinforcement learning is trial-and-error learning. Poolside maintains close to a million fully containerized code repositories each with its own test suite as a massive, diverse RL environment. The model writes code, executes it, gets deterministic feedback, and learns. This is why software turns out to be the ideal domain for RL scaling: it is deterministic enough to provide clear reward signals, but diverse enough to avoid model collapse.
Along the way, Eiso and Tim get into the weeds on frontier lab operations (over 4000 experimental runs per month), why Chinchilla optimality breaks down once you account for inference cost, what DeepSeek V3 tells us about the second-generation AI company playbook, and why the R1 zero-shot reasoning result should have been the real headline rather than the dollar figure. There is a candid exchange on whether the software development lifecycle itself will progressively collapse into the model, what Chris Olah interpretability work means for alignment, and whether Karpathy was right about Software 2.0 with some important caveats Eiso has developed after a decade of building AI for code.
SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.
TIMESTAMPS:
00:00:00 Introduction and Guest Background
00:02:50 Poolside AIs Vision and Three-Step Plan
00:06:50 Foundation Models vs. Enterprise Customization
00:10:25 The Missing Scaling Axis: Reinforcement Learning
00:15:40 Reinforcement Learning from Code Execution Feedback
00:22:20 Model Economics and Experimental Optimization
00:26:00 Enterprise Deployment Strategy and Market Focus
00:30:30 DeepSeek, Distributed Training, and Hardware Architecture
00:36:40 Emergent Reasoning and Chain-of-Thought Scaling
00:45:00 AI-Assisted Software Development Today
00:58:20 Architecture Innovation and Model Interpretability
01:15:00 Karpathys Software 2.0 and the Future of Code
01:25:00 AWS Partnership, Enterprise Security, and Closing
REFERENCES:
website:
[00:01:40] Tufa AI Labs
https://tufalabs.ai/
[00:02:50] Poolside AI
https://poolside.ai/
social:
[00:02:50] Eiso Kant on X
https://x.com/eisokant
paper:
[00:15:40] The Curse of Recursion: Training on Generated Data Makes Models Forget
https://arxiv.org/abs/2305.17493
[00:20:00] Efficient Estimation of Word Representations in Vector Space (Word2Vec)
https://arxiv.org/abs/1301.3781
[00:22:40] On the Measure of Intelligence
https://arxiv.org/abs/1911.01547
[00:30:30] DeepSeek-V3 Technical Report
https://arxiv.org/abs/2412.19437
[00:34:30] Training Compute-Optimal Large Language Models (Chinchilla)
https://arxiv.org/abs/2203.15556
[00:45:45] The Bitter Lesson
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
[00:49:55] Mastering the Game of Go with Deep Neural Networks and Tree Search
https://www.nature.com/articles/nature16961
[01:10:15] Mamba: Linear-Time Sequence Modeling with Selective State Spaces
https://arxiv.org/abs/2312.00752
[01:14:25] Zoom In: An Introduction to Circuits
https://distill.pub/2020/circuits/zoom-in/
benchmark:
[00:46:10] ARC Prize Challenge
https://arcprize.org/
blog:
[01:19:10] Software 2.0
https://karpathy.medium.com/software-2-0-a64152b37c35
LINKS:
Full Transcript: https://app.rescript.info/share/a8144052b0a5fc7210ab37f9651a3557
Download PDF transcript: https://app.rescript.info/api/public/sessions/2f5beb6d70f79973/pdf
Eiso Kant:
https://x.com/eisokant
https://poolside.ai/")
![Can Latent Program Networks Solve Abstract Reasoning? [Clement Bonnet]](https://i.ytimg.com/vi/PHBItVuudbU/mqdefault.jpg "Can Latent Program Networks Solve Abstract Reasoning? [Clement Bonnet]
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!
https://centml.ai/pricing/
Clement Bonnet presents Latent Program Networks (LPNs), a fundamentally different approach to the ARC-AGI benchmark that embeds programs into a continuous latent space rather than generating code or fine-tuning language models. The architecture uses a VAE-style encoder to map input-output pairs into distributional program representations, performs gradient-based search during both training and inference to refine these representations, and decodes solutions directly without an explicit program intermediary.
The conversation explores why ARC remains impervious to standard neural networks Chollet designed it to resist memorization, with test tasks far enough from any training distribution that zero-shot generalization fails. Bonnet and Tim Scarfe dig into the induction vs. transduction distinction, the role of the kernel trick in blurring that boundary, and why parameter-space search (test-time training) may be less efficient than searching a compressed program manifold.
A key architectural insight: training the search procedure end-to-end forces the latent space to become smooth and searchable, similar to MAML-style meta-learning. The system achieves ~10% on ARC evaluation sets using only small transformers (40M parameters total) trained from scratch on Michael Hodels re-ARC dataset no pre-trained LLMs, no internet priors.
The discussion also probes the limits of continuous latent spaces for compositional reasoning. Bonnet acknowledges that single-thread search through a fixed latent space cannot achieve true program composition, but suggests multi-threaded search with external composition could bridge the gap. The episode closes with reflections on creativity, the efficiency gap between human hypothesis-testing (a handful of guesses) and LLM program sampling (millions of samples), and what scaling LPNs might reveal about the structure of program spaces.
REFERENCES:
paper:
[00:00:05] ARC-AGI Benchmark
https://arxiv.org/abs/2412.04604
[00:02:10] Latent Program Networks
https://arxiv.org/abs/2411.08706
[00:08:45] Induction vs Transduction in Abstract Reasoning
https://arxiv.org/abs/2411.02272
[00:17:40] Variational Autoencoders
https://arxiv.org/abs/1312.6114
[00:33:00] Critique of Deep Learning
https://arxiv.org/abs/2002.06177
person:
[00:07:45] Kevin Ellis - Program Synthesis
https://www.cs.cornell.edu/~ellisk/
tool:
[00:27:50] re-ARC Framework
https://github.com/michaelhodel/re-arc
LINKS:
Full Transcript: https://app.rescript.info/share/55768bb71fbcc62c0522136650ccf960
Download PDF transcript: https://app.rescript.info/api/public/sessions/54ec5e9b3152760b/pdf
Clem and Matthew-
https://www.linkedin.com/in/clement-bonnet16/
https://github.com/clement-bonnet
https://mvmacfarlane.github.io/
[00:27:50] re-ARC, Hodel
https://github.com/michaelhodel/re-arc
[00:29:40] Grid size in ARC tasks, Chollet
https://github.com/fchollet/ARC-AGI")
![Mutually Assured AI Malfunction [Dan Hendrycks]](https://i.ytimg.com/vi/PM1waDBNDhw/mqdefault.jpg "Mutually Assured AI Malfunction [Dan Hendrycks]
Deep dive with Dan Hendrycks, a leading AI safety researcher and co-author of the Superintelligence Strategy paper with former Google CEO Eric Schmidt and Scale AI CEO Alexandr Wang.
*** SPONSOR MESSAGES
Gemini CLI is an open-source AI agent that brings the power of Gemini directly into your terminal - https://github.com/google-gemini/gemini-cli
Prolific: Quality data. From real people. For faster breakthroughs.
https://prolific.com/mlst?utm_campaign=98404559-MLST&utm_source=youtube&utm_medium=podcast&utm_content=script-gen
***
Hendrycks argues that society is making a fundamental mistake in how it views artificial intelligence. We often compare AI to transformative but ultimately manageable technologies like electricity or the internet. He contends a far better and more realistic analogy is nuclear technology. Like nuclear power, AI has the potential for immense good, but it is also a dual-use technology that carries the risk of unprecedented catastrophe.
The Problem with an AI Manhattan Project:
A popular idea is for the U.S. to launch a Manhattan Project for AI—a secret, all-out government race to build a superintelligence before rivals like China. Hendrycks argues this strategy is deeply flawed and dangerous for several reasons:
- It wouldn’t be secret. You cannot hide a massive, heat-generating data center from satellite surveillance.
- It would be destabilizing. A public race would alarm rivals, causing them to start their own desperate, corner-cutting projects, dramatically increasing global risk.
- It’s vulnerable to sabotage. An AI project can be crippled in many ways, from cyberattacks that poison its training data to physical attacks on its power plants. This is what the paper refers to as a maiming attack.
This vulnerability leads to the papers central concept: Mutual Assured AI Malfunction (MAIM). This is the AI-era version of the nuclear-eras Mutual Assured Destruction (MAD). In this dynamic, any nation that makes an aggressive, destabilizing bid for a world-dominating AI must expect its rivals to sabotage the project to ensure their own survival.
This deterrence, Hendrycks argues, is already the default reality we live in.
A Better Strategy: The Three Pillars
Instead of a reckless race, the paper proposes a more stable, three-part strategy modeled on Cold War principles:
- Deterrence: Acknowledge the reality of MAIM. The goal should not be to win the race to superintelligence, but to deter anyone from starting such a race in the first place through the credible threat of sabotage.
- Nonproliferation: Just as we work to keep fissile materials for nuclear bombs out of the hands of terrorists and rogue states, we must control the key inputs for catastrophic AI. The most critical input is advanced AI chips (GPUs). Hendrycks makes the powerful claim that building cutting-edge GPUs is now more difficult than enriching uranium, making this strategy viable.
- Competitiveness: The race between nations like the U.S. and China should not be about who builds superintelligence first. Instead, it should be about who can best use existing AI to build a stronger economy, a more effective military, and more resilient supply chains (for example, by manufacturing more chips domestically).
Dan says the stakes are high if we fail to manage this transition:
- Erosion of Control: Society becomes so dependent on AI systems for its economy and military that we can no longer turn them off without risking total collapse. We become passengers in an autonomous economy where humans are no longer in the drivers seat.
- Intelligence Recursion: This is the scenario where AI becomes capable of improving itself, kicking off a rapid intelligence explosion that could race ahead of any human attempts at control.
- Worthless Labor: When AI can perform most human cognitive tasks, the economic value of human labor could plummet, leading to massive societal instability and stripping people of their bargaining power.
Hendrycks maintains that while the risks are existential, the future is not set.
TOC:
1 Measuring the Beast [00:00:00]
2 Defining the Beast [00:11:34]
3 The Core Strategy [00:38:20]
4 Ideological Battlegrounds [00:53:12]
5 Mechanisms of Control [01:34:45]
TRANSCRIPT:
https://app.rescript.info/public/share/cOKcz4pWRPjh7BTIgybd7PUr_vChUaY6VQW64No8XMs
REFS:
Superintelligence Strategy
https://arxiv.org/abs/2503.05628
Humanitys Last Exam
https://arxiv.org/abs/2501.14249
Enigma Eval
https://arxiv.org/abs/2502.08859
Natural Selection Favors AIs Over Humans
https://arxiv.org/abs/2303.16200
Utility Engineering
https://arxiv.org/abs/2502.08640
Unsolved Problems in ML Safety [Emergence ref]
https://arxiv.org/abs/2109.13916
Situational Awareness by Leopold Aschenbrenner
https://situational-awareness.ai/
Large Language Models and Emergence [Krakauer]
https://arxiv.org/abs/2506.11135
Fractured Entangled Representations [Stanley/Kumar]
https://arxiv.org/pdf/2505.11581")
