Uploaded September 2024 | Updated September 2026, 1 week ago
Akbir Khan, AI researcher and ICML 2024 Best Paper winner, joins Tim Scarfe to discuss his groundbreaking work on using debate between language models to improve AI truthfulness and oversight. Khan explains how pitting two LLMs against each other in structured arguments helps non-expert judges arrive at more accurate answers than simply querying a single model — a result with profound implications for supervising AI systems that may eventually surpass human capabilities.
The conversation moves through the mechanics of scalable oversight and the "sandwiching protocol" that operationalises the problem of checking entities smarter than their supervisors. Khan describes how debate naturally surfaces the cruxes of disagreements, making complex expert judgments more accessible to laypeople. The discussion broadens into the relationship between intelligence and agency, the risks of deceptive alignment and reward tampering, and whether the current trajectory of AI development constitutes the early stages of a Cambrian explosion in artificial minds.
Khan and Scarfe also explore open-ended AI systems, Kenneth Stanley's arguments against objective-driven optimization, and the philosophical terrain mapped by thinkers like Aaron Sloman and Francois Chollet on the space of possible minds and measuring intelligence. The episode closes with a frank exchange on cultural evolution, memetics, and whether the intelligence that matters most for AI safety is the kind we can measure.
---
REFERENCES:
person:
[00:00:00] Akbir Khan
https://akbir.dev/
other:
[00:00:00] MLST Episode Shownotes PDF
dropbox.com/scl/fi/sjekivbg3ok6qugsv2p1u/AkbirKhan.pdf?rlkey=ewiyvq0aq7mjvql4u7os0jos2&st=vblhp7af&dl=0
[00:06:05] OpenAI Superalignment Team
openai.com/index/introducing-superalignment
[00:19:10] DeepMind Responsible AI
https://deepmind.google/about/responsibility-safety/
paper:
[00:00:40] Akbir Khan et al. - Debating with More Persuasive LLMs
arxiv.org/html/2402.06782v3
[00:08:10] Sam Bowman - Scalable Oversight in AI Systems
arxiv.org/abs/2211.03540
[00:10:35] Sam Bowman - Artificial Sandwiching Protocol
alignmentforum.org/posts/nekLYqbCEBDEfbLzF/artificial-sandwiching-when-can-we-test-scalable-alignment
[00:14:35] Janus - Simulators
lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators
[00:21:30] Eliezer Yudkowsky - AI FOOM Debate
intelligence.org/files/AIFoomDebate.pdf
[00:21:45] Sammy Martin - Discontinuous AI Progress
alignmentforum.org/posts/5WECpYABCT62TJrhY/will-ai-undergo-discontinuous-progress
[00:24:35] Nora Belrose - Counting Arguments vs AI Doom
lesswrong.com/posts/YsFZF3K9tuzbfrLxo/counting-arguments-provide-no-evidence-for-ai-doom
[00:25:35] Evan Hubinger - Deceptive Alignment
lesswrong.com/posts/zthDPAjh9w6Ytbeks/deceptive-alignment
[00:26:50] Anthropic - Reward Tampering
anthropic.com/research/reward-tampering
[00:34:58] Ryan Greenblatt et al. - AI Control
arxiv.org/pdf/2312.06942
[00:37:20] Aaron Sloman - The Space of Possible Minds
cs.bham.ac.uk/research/projects/cogaff/sloman-space-of-minds-84.pdf
[00:38:45] Francois Chollet - On the Measure of Intelligence
arxiv.org/abs/1911.01547
[00:42:45] Jonathan Cook et al. - Artificial Generational Intelligence
arxiv.org/abs/2406.00392
video:
[00:03:28] Yann LeCun on Machine Learning Debates
youtube.com/watch?v=OKkEdTchsiE
book:
[00:16:35] Thomas Suddendorf - The Gap
amazon.in/GAP-Science-Separates-Other-Animals/dp/0465030149
[00:32:35] Kenneth Stanley - Why Greatness Cannot Be Planned
amazon.co.uk/Why-Greatness-Cannot-Planned-Objective/dp/3319155237
[00:42:30] Richard Dawkins - The Selfish Gene
amazon.co.uk/Selfish-Gene-Richard-Dawkins/dp/0192860925
---
LINKS:
Full Transcript: app.rescript.info/share/c75b45c237ad52154d276851af8f812b
Download PDF transcript: app.rescript.info/api/public/sessions/571abed49dab3e7f/pdf
Akbir Khan:
https://x.com/akbirkhan
https://akbir.dev/
Akbir Khan, AI researcher and ICML 2024 Best Paper winner, joins Tim Scarfe to discuss his groundbreaking work on using debate between language models to improve AI truthfulness and oversight. Khan explains how pitting two LLMs against each other in structured arguments helps non-expert judges arrive at more accurate answers than simply querying a single model — a result with profound implications for supervising AI systems that may eventually surpass human capabilities.
The conversation moves through the mechanics of scalable oversight and the "sandwiching protocol" that operationalises the problem of checking entities smarter than their supervisors. Khan describes how debate naturally surfaces the cruxes of disagreements, making complex expert judgments more accessible to laypeople. The discussion broadens into the relationship between intelligence and agency, the risks of deceptive alignment and reward tampering, and whether the current trajectory of AI development constitutes the early stages of a Cambrian explosion in artificial minds.
Khan and Scarfe also explore open-ended AI systems, Kenneth Stanley's arguments against objective-driven optimization, and the philosophical terrain mapped by thinkers like Aaron Sloman and Francois Chollet on the space of possible minds and measuring intelligence. The episode closes with a frank exchange on cultural evolution, memetics, and whether the intelligence that matters most for AI safety is the kind we can measure.
---
REFERENCES:
person:
[00:00:00] Akbir Khan
https://akbir.dev/
other:
[00:00:00] MLST Episode Shownotes PDF
dropbox.com/scl/fi/sjekivbg3ok6qugsv2p1u/AkbirKhan.pdf?rlkey=ewiyvq0aq7mjvql4u7os0jos2&st=vblhp7af&dl=0
[00:06:05] OpenAI Superalignment Team
openai.com/index/introducing-superalignment
[00:19:10] DeepMind Responsible AI
https://deepmind.google/about/responsibility-safety/
paper:
[00:00:40] Akbir Khan et al. - Debating with More Persuasive LLMs
arxiv.org/html/2402.06782v3
[00:08:10] Sam Bowman - Scalable Oversight in AI Systems
arxiv.org/abs/2211.03540
[00:10:35] Sam Bowman - Artificial Sandwiching Protocol
alignmentforum.org/posts/nekLYqbCEBDEfbLzF/artificial-sandwiching-when-can-we-test-scalable-alignment
[00:14:35] Janus - Simulators
lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators
[00:21:30] Eliezer Yudkowsky - AI FOOM Debate
intelligence.org/files/AIFoomDebate.pdf
[00:21:45] Sammy Martin - Discontinuous AI Progress
alignmentforum.org/posts/5WECpYABCT62TJrhY/will-ai-undergo-discontinuous-progress
[00:24:35] Nora Belrose - Counting Arguments vs AI Doom
lesswrong.com/posts/YsFZF3K9tuzbfrLxo/counting-arguments-provide-no-evidence-for-ai-doom
[00:25:35] Evan Hubinger - Deceptive Alignment
lesswrong.com/posts/zthDPAjh9w6Ytbeks/deceptive-alignment
[00:26:50] Anthropic - Reward Tampering
anthropic.com/research/reward-tampering
[00:34:58] Ryan Greenblatt et al. - AI Control
arxiv.org/pdf/2312.06942
[00:37:20] Aaron Sloman - The Space of Possible Minds
cs.bham.ac.uk/research/projects/cogaff/sloman-space-of-minds-84.pdf
[00:38:45] Francois Chollet - On the Measure of Intelligence
arxiv.org/abs/1911.01547
[00:42:45] Jonathan Cook et al. - Artificial Generational Intelligence
arxiv.org/abs/2406.00392
video:
[00:03:28] Yann LeCun on Machine Learning Debates
youtube.com/watch?v=OKkEdTchsiE
book:
[00:16:35] Thomas Suddendorf - The Gap
amazon.in/GAP-Science-Separates-Other-Animals/dp/0465030149
[00:32:35] Kenneth Stanley - Why Greatness Cannot Be Planned
amazon.co.uk/Why-Greatness-Cannot-Planned-Objective/dp/3319155237
[00:42:30] Richard Dawkins - The Selfish Gene
amazon.co.uk/Selfish-Gene-Richard-Dawkins/dp/0192860925
---
LINKS:
Full Transcript: app.rescript.info/share/c75b45c237ad52154d276851af8f812b
Download PDF transcript: app.rescript.info/api/public/sessions/571abed49dab3e7f/pdf
Akbir Khan:
https://x.com/akbirkhan
https://akbir.dev/
![AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]](https://i.ytimg.com/vi/aoX_pGQMbEM/mqdefault.jpg "AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]
Dr. Ilia Shumailov - Former DeepMind AI Security Researcher, now building security tools for AI agents
Ever wondered what happens when AI agents start talking to each other—or worse, when they start breaking things? Ilia Shumailov spent years at DeepMind thinking about exactly these problems, and hes here to explain why securing AI is way harder than you think.
**SPONSOR MESSAGES**
—
Check out notebooklm for your research project, its really powerful
https://notebooklm.google.com/
—
Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
Were racing toward a world where AI agents will handle our emails, manage our finances, and interact with sensitive data 24/7. But there is a problem. These agents are nothing like human employees. They never sleep, they can touch every endpoint in your system simultaneously, and they can generate sophisticated hacking tools in seconds. Traditional security measures designed for humans simply wont work.
Dr. Ilia Shumailov
https://x.com/iliaishacked
https://iliaishacked.github.io/
https://sequrity.ai/
TRANSCRIPT:
https://app.rescript.info/public/share/dVGsk8dz9_V0J7xMlwguByBq1HXRD6i4uC5z5r7EVGM
More from Ilia on our Patreon:
https://www.patreon.com/posts/116142401/ (interview from last year)
https://www.patreon.com/posts/ilia-shumailov-140359158 (extended version of this interview)
TOC:
00:00:00 - Introduction & Trusted Third Parties via ML
00:03:45 - Background & Career Journey
00:06:42 - Safety vs Security Distinction
00:09:45 - Prompt Injection & Model Capability
00:13:00 - Agents as Worst-Case Adversaries
00:15:45 - Personal AI & CAML System Defense
00:19:30 - Agents vs Humans: Threat Modeling
00:22:30 - Calculator Analogy & Agent Behavior
00:25:00 - IMO Math Solutions & Agent Thinking
00:28:15 - Diffusion of Responsibility & Insider Threats
00:31:00 - Open Source Security Concerns
00:34:45 - Supply Chain Attacks & Trust Issues
00:39:45 - Architectural Backdoors
00:44:00 - Academic Incentives & Defense Work
00:48:30 - Semantic Censorship & Halting Problem
00:52:00 - Model Collapse: Theory & Criticism
00:59:30 - Career Advice & Ross Anderson Tribute
REFS:
Lessons from Defending Gemini Against Indirect Prompt Injections
https://arxiv.org/abs/2505.14534
Defeating Prompt Injections by Design. Google, Google DeepMind, and ETH Zurich. (CAML)
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F.
https://arxiv.org/pdf/2503.18813
Agentic Misalignment: How LLMs could be insider threats
https://www.anthropic.com/research/agentic-misalignment
STOP ANTHROPOMORPHIZING INTERMEDIATE TOKENS AS REASONING/THINKING TRACES!
Subbarao Kambhampati et al
https://arxiv.org/pdf/2504.09762
Meiklejohn, S., Blauzvern, H., Maruseac, M., Schrock, S., Simon, L., & Shumailov, I. (2025).
Machine learning models have a supply chain problem.
https://arxiv.org/abs/2505.22778
Gao, Y., Shumailov, I., & Fawaz, K. (2025).
Supply-chain attacks in machine learning frameworks.
In Proceedings of the 8th MLSys Conference.
https://openreview.net/pdf?id=EH5PZW6aCr
Apache Log4j Vulnerability Guidance
https://www.cisa.gov/news-events/news/apache-log4j-vulnerability-guidance
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2023).
Architectural backdoors in neural networks.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 21163–21173).
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2022).
Architectural backdoors in neural networks. arXiv preprint arXiv:2206.07840.
https://arxiv.org/pdf/2206.07840
Langford, H., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2024).
Architectural neural backdoors from first principles.
arXiv preprint arXiv:2402.06957.
Küchler, N., Petrov, I., Grobler, C., & Shumailov, I. (2025).
Architectural backdoors for within-batch data stealing and model inference manipulation.
arXiv preprint arXiv:2505.18323.
Position: Fundamental Limitations of LLM Censorship Necessitate New Approaches
David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Papernot, Vardan Papyan
https://proceedings.mlr.press/v235/glukhov24a.html
AlphaEvolve MLST interview [Matej Balog, Alexander Novikov]
https://www.youtube.com/watch?v=vC9nAosXrJw")
