![AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov] @MachineLearningStreetTalk](https://i.ytimg.com/vi/aoX_pGQMbEM/maxresdefault.jpg)
Uploaded October 2025 | Updated September 2026, 1 week ago
Dr. Ilia Shumailov - Former DeepMind AI Security Researcher, now building security tools for AI agents
Ever wondered what happens when AI agents start talking to each other—or worse, when they start breaking things? Ilia Shumailov spent years at DeepMind thinking about exactly these problems, and he's here to explain why securing AI is way harder than you think.
**SPONSOR MESSAGES**
—
Check out notebooklm for your research project, it's really powerful
notebooklm.google.com
—
Take the Prolific human data survey - prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
We're racing toward a world where AI agents will handle our emails, manage our finances, and interact with sensitive data 24/7. But there is a problem. These agents are nothing like human employees. They never sleep, they can touch every endpoint in your system simultaneously, and they can generate sophisticated hacking tools in seconds. Traditional security measures designed for humans simply won't work.
Dr. Ilia Shumailov
https://x.com/iliaishacked
iliaishacked.github.io
sequrity.ai
TRANSCRIPT:
app.rescript.info/public/share/dVGsk8dz9_V0J7xMlwguByBq1HXRD6i4uC5z5r7EVGM
More from Ilia on our Patreon:
patreon.com/posts/116142401 (interview from last year)
patreon.com/posts/ilia-shumailov-140359158 (extended version of this interview)
TOC:
00:00:00 - Introduction & Trusted Third Parties via ML
00:03:45 - Background & Career Journey
00:06:42 - Safety vs Security Distinction
00:09:45 - Prompt Injection & Model Capability
00:13:00 - Agents as Worst-Case Adversaries
00:15:45 - Personal AI & CAML System Defense
00:19:30 - Agents vs Humans: Threat Modeling
00:22:30 - Calculator Analogy & Agent Behavior
00:25:00 - IMO Math Solutions & Agent Thinking
00:28:15 - Diffusion of Responsibility & Insider Threats
00:31:00 - Open Source Security Concerns
00:34:45 - Supply Chain Attacks & Trust Issues
00:39:45 - Architectural Backdoors
00:44:00 - Academic Incentives & Defense Work
00:48:30 - Semantic Censorship & Halting Problem
00:52:00 - Model Collapse: Theory & Criticism
00:59:30 - Career Advice & Ross Anderson Tribute
REFS:
Lessons from Defending Gemini Against Indirect Prompt Injections
arxiv.org/abs/2505.14534
Defeating Prompt Injections by Design. Google, Google DeepMind, and ETH Zurich. (CAML)
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F.
arxiv.org/pdf/2503.18813
Agentic Misalignment: How LLMs could be insider threats
anthropic.com/research/agentic-misalignment
STOP ANTHROPOMORPHIZING INTERMEDIATE TOKENS AS REASONING/THINKING TRACES!
Subbarao Kambhampati et al
arxiv.org/pdf/2504.09762
Meiklejohn, S., Blauzvern, H., Maruseac, M., Schrock, S., Simon, L., & Shumailov, I. (2025).
Machine learning models have a supply chain problem.
arxiv.org/abs/2505.22778
Gao, Y., Shumailov, I., & Fawaz, K. (2025).
Supply-chain attacks in machine learning frameworks.
In Proceedings of the 8th MLSys Conference.
openreview.net/pdf?id=EH5PZW6aCr
Apache Log4j Vulnerability Guidance
cisa.gov/news-events/news/apache-log4j-vulnerability-guidance
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2023).
Architectural backdoors in neural networks.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 21163–21173).
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2022).
Architectural backdoors in neural networks. arXiv preprint arXiv:2206.07840.
arxiv.org/pdf/2206.07840
Langford, H., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2024).
Architectural neural backdoors from first principles.
arXiv preprint arXiv:2402.06957.
Küchler, N., Petrov, I., Grobler, C., & Shumailov, I. (2025).
Architectural backdoors for within-batch data stealing and model inference manipulation.
arXiv preprint arXiv:2505.18323.
Position: Fundamental Limitations of LLM Censorship Necessitate New Approaches
David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Papernot, Vardan Papyan
https://proceedings.mlr.press/v235/glukhov24a.html
AlphaEvolve MLST interview [Matej Balog, Alexander Novikov]
youtube.com/watch?v=vC9nAosXrJw
Dr. Ilia Shumailov - Former DeepMind AI Security Researcher, now building security tools for AI agents
Ever wondered what happens when AI agents start talking to each other—or worse, when they start breaking things? Ilia Shumailov spent years at DeepMind thinking about exactly these problems, and he's here to explain why securing AI is way harder than you think.
**SPONSOR MESSAGES**
—
Check out notebooklm for your research project, it's really powerful
notebooklm.google.com
—
Take the Prolific human data survey - prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
We're racing toward a world where AI agents will handle our emails, manage our finances, and interact with sensitive data 24/7. But there is a problem. These agents are nothing like human employees. They never sleep, they can touch every endpoint in your system simultaneously, and they can generate sophisticated hacking tools in seconds. Traditional security measures designed for humans simply won't work.
Dr. Ilia Shumailov
https://x.com/iliaishacked
iliaishacked.github.io
sequrity.ai
TRANSCRIPT:
app.rescript.info/public/share/dVGsk8dz9_V0J7xMlwguByBq1HXRD6i4uC5z5r7EVGM
More from Ilia on our Patreon:
patreon.com/posts/116142401 (interview from last year)
patreon.com/posts/ilia-shumailov-140359158 (extended version of this interview)
TOC:
00:00:00 - Introduction & Trusted Third Parties via ML
00:03:45 - Background & Career Journey
00:06:42 - Safety vs Security Distinction
00:09:45 - Prompt Injection & Model Capability
00:13:00 - Agents as Worst-Case Adversaries
00:15:45 - Personal AI & CAML System Defense
00:19:30 - Agents vs Humans: Threat Modeling
00:22:30 - Calculator Analogy & Agent Behavior
00:25:00 - IMO Math Solutions & Agent Thinking
00:28:15 - Diffusion of Responsibility & Insider Threats
00:31:00 - Open Source Security Concerns
00:34:45 - Supply Chain Attacks & Trust Issues
00:39:45 - Architectural Backdoors
00:44:00 - Academic Incentives & Defense Work
00:48:30 - Semantic Censorship & Halting Problem
00:52:00 - Model Collapse: Theory & Criticism
00:59:30 - Career Advice & Ross Anderson Tribute
REFS:
Lessons from Defending Gemini Against Indirect Prompt Injections
arxiv.org/abs/2505.14534
Defeating Prompt Injections by Design. Google, Google DeepMind, and ETH Zurich. (CAML)
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F.
arxiv.org/pdf/2503.18813
Agentic Misalignment: How LLMs could be insider threats
anthropic.com/research/agentic-misalignment
STOP ANTHROPOMORPHIZING INTERMEDIATE TOKENS AS REASONING/THINKING TRACES!
Subbarao Kambhampati et al
arxiv.org/pdf/2504.09762
Meiklejohn, S., Blauzvern, H., Maruseac, M., Schrock, S., Simon, L., & Shumailov, I. (2025).
Machine learning models have a supply chain problem.
arxiv.org/abs/2505.22778
Gao, Y., Shumailov, I., & Fawaz, K. (2025).
Supply-chain attacks in machine learning frameworks.
In Proceedings of the 8th MLSys Conference.
openreview.net/pdf?id=EH5PZW6aCr
Apache Log4j Vulnerability Guidance
cisa.gov/news-events/news/apache-log4j-vulnerability-guidance
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2023).
Architectural backdoors in neural networks.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 21163–21173).
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2022).
Architectural backdoors in neural networks. arXiv preprint arXiv:2206.07840.
arxiv.org/pdf/2206.07840
Langford, H., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2024).
Architectural neural backdoors from first principles.
arXiv preprint arXiv:2402.06957.
Küchler, N., Petrov, I., Grobler, C., & Shumailov, I. (2025).
Architectural backdoors for within-batch data stealing and model inference manipulation.
arXiv preprint arXiv:2505.18323.
Position: Fundamental Limitations of LLM Censorship Necessitate New Approaches
David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Papernot, Vardan Papyan
https://proceedings.mlr.press/v235/glukhov24a.html
AlphaEvolve MLST interview [Matej Balog, Alexander Novikov]
youtube.com/watch?v=vC9nAosXrJw
![AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]](https://i.ytimg.com/vi/aoX_pGQMbEM/mqdefault.jpg "AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]
Dr. Ilia Shumailov - Former DeepMind AI Security Researcher, now building security tools for AI agents
Ever wondered what happens when AI agents start talking to each other—or worse, when they start breaking things? Ilia Shumailov spent years at DeepMind thinking about exactly these problems, and hes here to explain why securing AI is way harder than you think.
**SPONSOR MESSAGES**
—
Check out notebooklm for your research project, its really powerful
https://notebooklm.google.com/
—
Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
Were racing toward a world where AI agents will handle our emails, manage our finances, and interact with sensitive data 24/7. But there is a problem. These agents are nothing like human employees. They never sleep, they can touch every endpoint in your system simultaneously, and they can generate sophisticated hacking tools in seconds. Traditional security measures designed for humans simply wont work.
Dr. Ilia Shumailov
https://x.com/iliaishacked
https://iliaishacked.github.io/
https://sequrity.ai/
TRANSCRIPT:
https://app.rescript.info/public/share/dVGsk8dz9_V0J7xMlwguByBq1HXRD6i4uC5z5r7EVGM
More from Ilia on our Patreon:
https://www.patreon.com/posts/116142401/ (interview from last year)
https://www.patreon.com/posts/ilia-shumailov-140359158 (extended version of this interview)
TOC:
00:00:00 - Introduction & Trusted Third Parties via ML
00:03:45 - Background & Career Journey
00:06:42 - Safety vs Security Distinction
00:09:45 - Prompt Injection & Model Capability
00:13:00 - Agents as Worst-Case Adversaries
00:15:45 - Personal AI & CAML System Defense
00:19:30 - Agents vs Humans: Threat Modeling
00:22:30 - Calculator Analogy & Agent Behavior
00:25:00 - IMO Math Solutions & Agent Thinking
00:28:15 - Diffusion of Responsibility & Insider Threats
00:31:00 - Open Source Security Concerns
00:34:45 - Supply Chain Attacks & Trust Issues
00:39:45 - Architectural Backdoors
00:44:00 - Academic Incentives & Defense Work
00:48:30 - Semantic Censorship & Halting Problem
00:52:00 - Model Collapse: Theory & Criticism
00:59:30 - Career Advice & Ross Anderson Tribute
REFS:
Lessons from Defending Gemini Against Indirect Prompt Injections
https://arxiv.org/abs/2505.14534
Defeating Prompt Injections by Design. Google, Google DeepMind, and ETH Zurich. (CAML)
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F.
https://arxiv.org/pdf/2503.18813
Agentic Misalignment: How LLMs could be insider threats
https://www.anthropic.com/research/agentic-misalignment
STOP ANTHROPOMORPHIZING INTERMEDIATE TOKENS AS REASONING/THINKING TRACES!
Subbarao Kambhampati et al
https://arxiv.org/pdf/2504.09762
Meiklejohn, S., Blauzvern, H., Maruseac, M., Schrock, S., Simon, L., & Shumailov, I. (2025).
Machine learning models have a supply chain problem.
https://arxiv.org/abs/2505.22778
Gao, Y., Shumailov, I., & Fawaz, K. (2025).
Supply-chain attacks in machine learning frameworks.
In Proceedings of the 8th MLSys Conference.
https://openreview.net/pdf?id=EH5PZW6aCr
Apache Log4j Vulnerability Guidance
https://www.cisa.gov/news-events/news/apache-log4j-vulnerability-guidance
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2023).
Architectural backdoors in neural networks.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 21163–21173).
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2022).
Architectural backdoors in neural networks. arXiv preprint arXiv:2206.07840.
https://arxiv.org/pdf/2206.07840
Langford, H., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2024).
Architectural neural backdoors from first principles.
arXiv preprint arXiv:2402.06957.
Küchler, N., Petrov, I., Grobler, C., & Shumailov, I. (2025).
Architectural backdoors for within-batch data stealing and model inference manipulation.
arXiv preprint arXiv:2505.18323.
Position: Fundamental Limitations of LLM Censorship Necessitate New Approaches
David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Papernot, Vardan Papyan
https://proceedings.mlr.press/v235/glukhov24a.html
AlphaEvolve MLST interview [Matej Balog, Alexander Novikov]
https://www.youtube.com/watch?v=vC9nAosXrJw")
![The Weird ChatGPT Hack That Leaked Training Data [Dr. Yannic Kilcher / Prof. Florian Tramer]](https://i.ytimg.com/vi/c_hmxRVDXBE/mqdefault.jpg "The Weird ChatGPT Hack That Leaked Training Data [Dr. Yannic Kilcher / Prof. Florian Tramer]
Dr. Yannic lightspeed Kilcher sits down with an AI Security Researcher Prof. Florian Tramer (an expert in the field, collaborating with folks like Nicholas Carlini and Milad Nasr at Google) to ruminate on the murky state of affairs in AI security.
Out of the gate they assert that AI detectors (those tools that try to spot if text is human-written or AI-generated) are basically doomed to fail—like trying to catch an eel with greasy hands blindfolded. Lightspeed Kilcher shares a funny story about how the word Delve kept rearing its ugly head in AI text because of training data emanating from Nigerian crowd workers, leading to false flags in job applications.
Tramer then takes the gloves off about security risks, saying that even a 99% success rate in AI means total failure in security terms—attackers will always find that 1% weak spot. They share an interesting example of a weird attack on ChatGPT: asking it to repeat poem forever made it spit out memorized internet scraps, leaking training data like a sieve. OpenAI patched it, but its like slapping a band-aid on a shark bite.
Lightspeed Kilcher asks about top worries, and Tramer lists three big ones in relatable terms: 1) Training AI on private data (like medical records) could lead to massive leaks, since models memorize stuff without us fully understanding how; 2) Prompt injections, where bad actors hijack AI agents to do harmful things, like Anthropics demo that controls computers—its like giving your keys to a stranger; and 3) The rush to deploy AI without safeguards, echoing past tech blunders like SQL hacks.
They reflect on how ChatGPT has lit a fire under AI security research—making it amazing and scary by turning hypotheticals into real-world issues with millions of users. The conversation touches on why just scaling up AI wont fix everything (we need causal understanding, not just more data), and why watermarking AI outputs isnt a silver bullet—its easy to bypass, especially with open-source models.
Its about time we had our old comrade light speed Kilcher back on MLST!
TRANSCRIPT:
https://app.rescript.info/public/share/PGieXFZCRyJjJswicatmNFia_pKCt_WVP-pZS4qguiw
LINKS:
Scalable Extraction of Training Data from (Production) Language Models [Poem Attack]
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel
https://arxiv.org/abs/2311.17035
- Prof. Florian Tramer
https://www.floriantramer.com/
- Yannic Kilchers YouTube channel
https://www.youtube.com/c/YannicKilcher
- Nicholas Carlinis research profile (collaborator on the data recovery project):
https://nicholas.carlini.com/ – Includes papers on AI attacks and memorization risks.")
![AI training data will never be fully synthetic [SPONSORED]](https://i.ytimg.com/vi/cnxZZTl1tkk/mqdefault.jpg "AI training data will never be fully synthetic [SPONSORED]
We sat down with Sara Saab (VP of Product at Prolific) and Enzo Blindow (VP of Data and AI at Prolific) to explore the critical role of human evaluation in AI development and the challenges of aligning AI systems with human values. Prolific is a human annotation and orchestration platform for AI used by many of the major AI labs. This is a sponsored show in partnership with Prolific.
**SPONSOR MESSAGES**
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
While technologists want to remove humans from the loop for speed and efficiency, these non-deterministic AI systems actually require more human oversight than ever before. Prolifics approach is to put well-treated, verified, diversely demographic humans behind an API - making human feedback as accessible as any other infrastructure service.
When AI models like Grok 4 achieve top scores on technical benchmarks but feel awkward or problematic to use in practice, it exposes the limitations of our current evaluation methods. The guests argue that optimizing for benchmarks may actually weaken model performance in other crucial areas, like cultural sensitivity or natural conversation.
We also discuss Anthropics research showing that frontier AI models, when given goals and access to information, independently arrived at solutions involving blackmail - without any prompting toward unethical behavior. Even more concerning, the more sophisticated the model, the more susceptible it was to this agentic misalignment.
Enzo and Sarah present Prolifics Humane leaderboard as an alternative to existing benchmarking systems. By stratifying evaluations across diverse demographic groups, they reveal that different populations have vastly different experiences with the same AI models.
Looking forwards the guests imagine a world where humans take on coaching and teaching roles for AI systems - similar to how we might correct a child or review code. Working conditions and the evolution of labor in an AI-augmented world is also important to consider. Rather than replacing humans entirely, we may be moving toward more sophisticated forms of human-AI collaboration.
We need more representative evaluation frameworks that capture the messy reality of human values and cultural diversity.
Visit Prolific:
https://www.prolific.com/
Sara Saab (VP Product):
https://uk.linkedin.com/in/sarasaab
Enzo Blindow (VP Data & AI):
https://uk.linkedin.com/in/enzoblindow
TRANSCRIPT:
https://app.rescript.info/public/share/xZ31-0kJJ_xp4zFSC-bunC8-hJNkHpbm7Lg88RFcuLE
TOC:
[00:00:00] Intro & Background
[00:03:16] Human-in-the-Loop Challenges
[00:17:19] Can AIs Understand?
[00:32:02] Benchmarking & Vibes
[00:51:00] Agentic Misalignment Study
[01:03:00] Data Quality vs Quantity
[01:16:00] Future of AI Oversight
REFS:
Anthropic Agentic Misalignment
https://www.anthropic.com/research/agentic-misalignment
Value Compass
https://arxiv.org/pdf/2409.09586
Reasoning Models Don’t Always Say What They Think (Anthropic)
https://www.anthropic.com/research/reasoning-models-dont-say-think
https://assets.anthropic.com/m/71876fabef0f0ed4/original/reasoning_models_paper.pdf
Maslow’s Hierarchy Of Needs
https://www.simplypsychology.org/maslow.html
Apollo research - science of evals blog post
https://www.apolloresearch.ai/blog/we-need-a-science-of-evals
Leaderboard Illusion
https://www.youtube.com/watch?v=9W_OhS38rIE MLST video
The Leaderboard Illusion [2025]
Shivalika Singh, Yiyang Nan, Alex Wang, Daniel DSouza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, Sara Hooker
https://arxiv.org/abs/2504.20879
Humanities last exam
https://arxiv.org/abs/2501.14249
PRISM paper
https://arxiv.org/abs/2405.10254
https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback
Collective intelligence project
https://www.cip.org/whitepaper
https://ischool.utoronto.ca/faculty-profile/brian-cantwell-smith/
Ghost work (Mary Gray)
https://www.amazon.com/Ghost-Work-Silicon-Building-Underclass/dp/1328566242
Fairwork Cloudwork report
https://fair.work/en/ratings/cloudwork/
Gibson theory of affordances
https://cs.brown.edu/courses/cs137/2017/readings/Gibson-AFF.pdf")
![We Built Calculators Because Were STUPID! [Prof. David Krakauer]](https://i.ytimg.com/vi/dY46YsGWMIc/mqdefault.jpg "We Built Calculators Because Were STUPID! [Prof. David Krakauer]
Prof. David Krakauer argues that intelligence isnt about knowing more—its about doing more with less.
His core thesis flips our assumptions on their head: while emergence in complex systems is about more is different, intelligence is fundamentally about less is more.
Why Large Language Models Miss the Point
LLMs are more is more, Krakauer explains. Theyre essentially sophisticated libraries that know everything but understand nothing. When a student looks up answers in a library during an exam, we call them a cheater, not intelligent.
Intelligence Across All Life
Perhaps most surprisingly, Krakauer claims that all living things are intelligent—even bacteria. His reasoning: intelligence is the information accumulated through evolutions selective pressures. Its why we can say an elephant is smarter than a worm, but wed never say its more alive. The difference is accumulated capacity.
Three Dimensions of Intelligence
Krakauer proposes intelligence exists in three distinct flavours:
- Strategic Intelligence - Adaptation and survival where viruses actually beat us/
- Inferential Intelligence - Math and computation (where we built calculators because were so bad at it)
- Representational Intelligence - Finding better ways to encode problems (the most uniquely human)
The Soma Cube Insight
Using a Soma Cube puzzle, Krakauer demonstrates something incredible: a four-year-old can solve a combinatorial problem with 15,000 logical clauses—not by computing, but by using physical representation. The physical world does the computing for us. This is intelligence: making impossibly hard problems tractable through clever representation.
This talk was recorded at DISI 2025 https://disi.org/
David Krakauer
President and William H. Miller Professor of Complex Systems
https://www.santafe.edu/people/profile/david-krakauer
Note this version is edited down, you can watch the full version on our Patreon - https://www.patreon.com/posts/masterclass-on-142898847
Watch interview we published with David - https://www.youtube.com/watch?v=jXa8dHzgV8U
TRANSCRIPT:
https://app.rescript.info/public/share/Cs8QFwMw_FiOEROkNPiIf6Y-e_ahAxNMl0oFiL7_4QA
TOC:
00:00:00 - Complexity, Life & Intelligence
00:00:45 - Ptolemy vs Newton
00:01:30 - Purpose of Science
00:02:15 - Feynman on Rules vs Strategies
00:03:00 - Entropy as Unifying Rule
00:03:45 - Intelligence in Physics
00:04:15 - Complex Systems & Broken Symmetries
00:05:00 - Historical Definitions of Intelligence
00:06:15 - Herbert: Capacity to Acquire Capacity
00:06:45 - Problems with Turing Test
00:07:30 - Intelligence vs Knowledge
00:08:00 - LLMs as Libraries
00:08:30 - Humanistic Perspectives
00:09:00 - Unified Theory: Life & Intelligence
00:09:30 - Intensive vs Extensive Properties
00:10:30 - Selection Gradient Analogy
00:11:15 - Universal Intelligence (Bacteria to Elephants)
00:12:30 - Three Dimensions of Intelligence
00:13:00 - Strategic Intelligence & Viruses
00:13:30 - Inferential Intelligence & Calculators
00:14:15 - Encoding & Representation
00:14:30 - Different Paths: Life vs AI
00:15:15 - Soma Cube Problem
00:16:00 - Principle of Materiality
00:17:15 - Embodied Representation
00:17:45 - Intelligence Makes Hard Problems Easy
00:18:00 - Stupidity & Conclusion
REFS:
Papers and Academic Works
More is Different by Phil Anderson [https://www.tkm.kit.edu/downloads/TKM1_2011_more_is_different_PWA.pdf ]
Recent paper with Melanie and John Krakauer on emergence [https://arxiv.org/abs/2506.11135 ]
Formal treatment paper on intelligence/evolution duality from David - Darwinian demons, evolutionary complexity, and information maximization [https://wiki.santafe.edu/images/b/b1/CHAOEH213037110_1.pdf ]
The Unreasonable Effectiveness of Mathematics in the Natural Sciences by Eugene Wigner [https://webhomes.maths.ed.ac.uk/~v1ranick/papers/wigner.pdf ]")
