Uploaded October 2024 | Updated November 2026, 1 day ago
MLST is sponsored by Tufa Labs:
Are you interested in working on ARC and cutting-edge AI research with the MindsAI team (current ARC winners)?
Focus: ARC, LLMs, test-time-compute, active inference, system2 reasoning, and more.
Future plans: Expanding to complex environments like Warcraft 2 and Starcraft 2.
Interested? Apply for an ML research position: benjamin@tufa.ai
Professor Alexander Ororbia from the Rochester Institute of Technology takes Tim Scarfe through the case for bio-inspired AI. The central idea is mortal computation: you cannot divorce the software from the hardware that runs it. The brain manages remarkable things on a few watts because its computations are entangled with its physical substrate. GPT-class models, running on von Neumann architectures designed for immortal computation -- where software and hardware are deliberately decoupled -- pay a staggering energy penalty for that separation.
Ororbia explains the building blocks: Markov blankets as the formalism for system boundaries, Karl Friston's free energy principle as the optimization target, and the MILLS framework (Mortal Inference, Learning, and Selection) operating across multiple timescales. He then surveys the landscape of alternatives to backpropagation -- predictive coding, Hebbian learning, contrastive methods, and Geoff Hinton's forward-forward algorithm -- showing how each maps to observations from neuroscience.
The conversation gets practical with Ororbia's ngc-learn library for implementing these algorithms, the stability-plasticity dilemma in continual learning, and the current state of neuromorphic hardware from Intel Loihi to IBM TrueNorth. He closes with his neural generative coding work, which showed that predictive coding networks can synthesize data they were never trained on -- outperforming VAEs and GANs -- and his vision for bio-inspired AI systems that coexist with humanity rather than replacing it.
---
TIMESTAMPS:
00:00:00 Introduction to Bio-Inspired AI and Mortal Computation
00:04:50 Principles of Mortal Computation
00:17:41 Markov Blankets and Free Energy Principle
00:24:38 MILLS Framework and Biological Systems
00:31:00 Challenging Backpropagation: Alternative Approaches
00:31:49 Predictive Coding and Free Energy Principle
00:41:52 Biologically Plausible Credit Assignment Methods
00:50:11 Taxonomy of Bio-inspired Learning Algorithms
00:59:30 Forward-Only Learning and ngc-learn Implementation
01:03:25 Stability-Plasticity Dilemma and Continual Learning
01:09:00 Neuromorphic Hardware and Challenges
01:12:58 Neural Generative Coding and Future Directions
---
REFERENCES:
website:
[00:04:43] The Levin Lab
drmichaellevin.org
[00:18:20] Good Regulator Theorem
en.wikipedia.org/wiki/Good_regulator
[00:41:52] Hebbian Theory
en.wikipedia.org/wiki/Hebbian_theory
[00:45:00] Hopfield Network
en.wikipedia.org/wiki/Hopfield_network
[01:09:00] Intel Loihi 2
intel.com/content/www/us/en/research/neuromorphic-computing-loihi-2-technology-brief.html
paper:
[00:04:50] Mortal Computation: A Foundation for Biomimetic Intelligence
arxiv.org/abs/2311.09589
[00:06:53] The Forward-Forward Algorithm
arxiv.org/abs/2212.13345
[00:07:20] There's Plenty of Room Right Here
ncbi.nlm.nih.gov/pmc/articles/PMC10046700
[00:17:41] The Free-Energy Principle: A Rough Guide to the Brain
fil.ion.ucl.ac.uk/~karl/The%20free-energy%20principle%20-%20a%20rough%20guide%20to%20the%20brain.pdf
[00:31:49] Predictive Coding in the Visual Cortex
nature.com/articles/nn0199_79
[00:41:52] Brain-Inspired Machine Intelligence: Neurobiologically-Plausible Credit Assignment
arxiv.org/abs/2312.09257
[00:45:50] A Tutorial on Energy-Based Learning
yann.lecun.com/exdb/publis/pdf/lecun-06.pdf
[00:46:40] A Learning Algorithm for Boltzmann Machines
https://www.cs.toronto.edu/~hinton/absps/cogscibm.pdf
[00:50:11] A Review of Neuroscience-Inspired Machine Learning
arxiv.org/abs/2403.18929
[00:53:20] NEAT: NeuroEvolution of Augmenting Topologies
https://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf
[00:56:40] A Path Towards Autonomous Machine Intelligence
openreview.net/pdf?id=BZ5a1r-kVsf
[00:59:30] Test-Time Model Adaptation with Only Forward Passes
arxiv.org/abs/2404.01650
[01:03:25] Spiking Neural Predictive Coding for Continual Learning
sciencedirect.com/science/article/pii/S0925231223004150
[01:10:00] IBM TrueNorth
research.ibm.com/publications/truenorth-design-and-tool-flow-of-a-65-mw-1-million-neuron-programmable-neurosynaptic-chip
book:
[00:24:38] Active Inference: The Free Energy Principle in Mind, Brain, and Behavior
https://direct.mit.edu/books/oa-monograph/5299/Active-InferenceThe-Free-Energy-Principle-in-Mind
---
LINKS:
Full Transcript: app.rescript.info/share/cfa14d0f39f86d035d5caf6173d6207f
Download PDF transcript: app.rescript.info/api/public/sessions/94d5c32d7c507e6a/pdf
MLST is sponsored by Tufa Labs:
Are you interested in working on ARC and cutting-edge AI research with the MindsAI team (current ARC winners)?
Focus: ARC, LLMs, test-time-compute, active inference, system2 reasoning, and more.
Future plans: Expanding to complex environments like Warcraft 2 and Starcraft 2.
Interested? Apply for an ML research position: benjamin@tufa.ai
Professor Alexander Ororbia from the Rochester Institute of Technology takes Tim Scarfe through the case for bio-inspired AI. The central idea is mortal computation: you cannot divorce the software from the hardware that runs it. The brain manages remarkable things on a few watts because its computations are entangled with its physical substrate. GPT-class models, running on von Neumann architectures designed for immortal computation -- where software and hardware are deliberately decoupled -- pay a staggering energy penalty for that separation.
Ororbia explains the building blocks: Markov blankets as the formalism for system boundaries, Karl Friston's free energy principle as the optimization target, and the MILLS framework (Mortal Inference, Learning, and Selection) operating across multiple timescales. He then surveys the landscape of alternatives to backpropagation -- predictive coding, Hebbian learning, contrastive methods, and Geoff Hinton's forward-forward algorithm -- showing how each maps to observations from neuroscience.
The conversation gets practical with Ororbia's ngc-learn library for implementing these algorithms, the stability-plasticity dilemma in continual learning, and the current state of neuromorphic hardware from Intel Loihi to IBM TrueNorth. He closes with his neural generative coding work, which showed that predictive coding networks can synthesize data they were never trained on -- outperforming VAEs and GANs -- and his vision for bio-inspired AI systems that coexist with humanity rather than replacing it.
---
TIMESTAMPS:
00:00:00 Introduction to Bio-Inspired AI and Mortal Computation
00:04:50 Principles of Mortal Computation
00:17:41 Markov Blankets and Free Energy Principle
00:24:38 MILLS Framework and Biological Systems
00:31:00 Challenging Backpropagation: Alternative Approaches
00:31:49 Predictive Coding and Free Energy Principle
00:41:52 Biologically Plausible Credit Assignment Methods
00:50:11 Taxonomy of Bio-inspired Learning Algorithms
00:59:30 Forward-Only Learning and ngc-learn Implementation
01:03:25 Stability-Plasticity Dilemma and Continual Learning
01:09:00 Neuromorphic Hardware and Challenges
01:12:58 Neural Generative Coding and Future Directions
---
REFERENCES:
website:
[00:04:43] The Levin Lab
drmichaellevin.org
[00:18:20] Good Regulator Theorem
en.wikipedia.org/wiki/Good_regulator
[00:41:52] Hebbian Theory
en.wikipedia.org/wiki/Hebbian_theory
[00:45:00] Hopfield Network
en.wikipedia.org/wiki/Hopfield_network
[01:09:00] Intel Loihi 2
intel.com/content/www/us/en/research/neuromorphic-computing-loihi-2-technology-brief.html
paper:
[00:04:50] Mortal Computation: A Foundation for Biomimetic Intelligence
arxiv.org/abs/2311.09589
[00:06:53] The Forward-Forward Algorithm
arxiv.org/abs/2212.13345
[00:07:20] There's Plenty of Room Right Here
ncbi.nlm.nih.gov/pmc/articles/PMC10046700
[00:17:41] The Free-Energy Principle: A Rough Guide to the Brain
fil.ion.ucl.ac.uk/~karl/The%20free-energy%20principle%20-%20a%20rough%20guide%20to%20the%20brain.pdf
[00:31:49] Predictive Coding in the Visual Cortex
nature.com/articles/nn0199_79
[00:41:52] Brain-Inspired Machine Intelligence: Neurobiologically-Plausible Credit Assignment
arxiv.org/abs/2312.09257
[00:45:50] A Tutorial on Energy-Based Learning
yann.lecun.com/exdb/publis/pdf/lecun-06.pdf
[00:46:40] A Learning Algorithm for Boltzmann Machines
https://www.cs.toronto.edu/~hinton/absps/cogscibm.pdf
[00:50:11] A Review of Neuroscience-Inspired Machine Learning
arxiv.org/abs/2403.18929
[00:53:20] NEAT: NeuroEvolution of Augmenting Topologies
https://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf
[00:56:40] A Path Towards Autonomous Machine Intelligence
openreview.net/pdf?id=BZ5a1r-kVsf
[00:59:30] Test-Time Model Adaptation with Only Forward Passes
arxiv.org/abs/2404.01650
[01:03:25] Spiking Neural Predictive Coding for Continual Learning
sciencedirect.com/science/article/pii/S0925231223004150
[01:10:00] IBM TrueNorth
research.ibm.com/publications/truenorth-design-and-tool-flow-of-a-65-mw-1-million-neuron-programmable-neurosynaptic-chip
book:
[00:24:38] Active Inference: The Free Energy Principle in Mind, Brain, and Behavior
https://direct.mit.edu/books/oa-monograph/5299/Active-InferenceThe-Free-Energy-Principle-in-Mind
---
LINKS:
Full Transcript: app.rescript.info/share/cfa14d0f39f86d035d5caf6173d6207f
Download PDF transcript: app.rescript.info/api/public/sessions/94d5c32d7c507e6a/pdf

![The Programming Language That Referees Mathematics — Leo de Moura
Leonardo de Moura created Lean and co-created Z3.
This episode is sponsored by Parallel.
Parallel, where agents find answers: web search, extraction and deep research APIs built for AI agents.
Start free with the Parallel MCP server and $5 of credits every month: https://parallel.ai/mlst?utm_source=creator&utm_medium=podcast&utm_content=MLST
Tim Scarfe talks with Leo about how Lean escaped its original audience, why dependent types and Mathlib made it useful to working mathematicians, and what happens when formal verification leaves the lab. De Moura explains the small trusted kernel and independent checkers, and gives his account of the recent Collatz incident, in which a purported proof was accepted by both Leans official kernel and Nanoda, apparently by exploiting a different bug in each.
The conversation then turns to AI-produced proofs and software. Claude agents rebuild zlib in Lean and the result is verified, yet the example exposes the loophole at the centre of the show: a proof only certifies the specification humans chose to write. De Moura describes proof search as a game, separates formal competence from mathematical understanding, and asks what authorship means when models can generate proofs at scale.
The closing argument puts human responsibility upstream: choosing definitions, judging abstractions, curating Mathlib, inspecting certificates, and deciding which problems deserve effort. He also reflects on Leans future beyond its founder and gives a practical starting point for new users.
TIMESTAMPS:
00:00:00 Cold open: the green checkmark can lie
00:00:59 Cathedral or bazaar: who controls Leans core?
00:04:55 Why Leans core stays small and protected
00:08:07 The Slack purge, Brandolinis law and the Lean FRO
00:11:12 The Collatz exploit: two kernels, two bugs
00:16:44 More kernels, reward hacking and safety by transparency
00:21:04 Sponsor: Parallel
00:21:59 Kim Morrison, Claude and the zlib proof
00:25:25 Can we specify complex systems?
00:28:04 Specs change: proofs are cheaper to redo with AI
00:31:27 From Lean 1 to Lean 4
00:34:57 Dependent types in plain terms
00:37:25 Lean 4s extensibility and Mathlibs growth
00:41:44 Mathlib as infrastructure: Formal Frontiers
00:44:15 Creativity, abstraction and nut-sniping
00:48:46 Breadcrumbs, not learning: what AI agents lack
00:53:03 Competence without comprehension, and verified guardrails
00:56:21 Is the human still the author?
01:01:44 AlphaProof, LLMs and why certificates still matter
01:06:38 Whats next for Lean, and its legacy
01:11:42 How to start learning Lean
REFERENCES:
tool:
[00:00:48] Lean
https://lean-lang.org/
[00:01:24] Mathlib
https://github.com/leanprover-community/mathlib4
[00:12:09] nanoda_lib
https://github.com/ammkrn/nanoda_lib
[00:12:19] CollatzLean
https://github.com/xrchz/CollatzLean/blob/a79357462a33d2a6babd4cf6c8d8bcd25425d653/README.md
[00:13:06] Lean issue 14576
https://github.com/leanprover/lean4/issues/14576
[00:13:21] Lean pull request 14577
https://github.com/leanprover/lean4/pull/14577
[00:13:45] nanoda_lib pull request 22
https://github.com/ammkrn/nanoda_lib/pull/22
[00:15:33] Lean comparator
https://github.com/leanprover/comparator
[00:18:02] Lean4Lean
https://github.com/digama0/lean4lean
[00:19:50] ARC-AGI-3
https://arcprize.org/arc-agi/3
[00:21:59] lean-zip
https://github.com/kim-em/lean-zip
[00:29:45] CompCert
https://compcert.org/
[00:29:45] seL4
https://www.sel4.org/
[00:30:23] Z3
https://github.com/Z3Prover/z3
[00:38:10] Veil
https://github.com/verse-lab/veil
[00:38:10] Velvet
https://github.com/verse-lab/velvet
organization:
[00:00:52] Lean FRO
https://lean-lang.org/fro/
[00:42:39] Mathlib Initiative
https://mathlib-initiative.org/about/
[01:11:42] Computerphile
https://www.youtube.com/@Computerphile
person:
[00:05:11] Ilya Sergey
https://ilyasergey.net/
[00:10:31] Joachim Breitner
https://www.joachim-breitner.de/
[00:32:58] Adam Chlipala
https://adam.chlipala.net/
[00:38:42] Kevin Buzzard
https://www.ma.imperial.ac.uk/~buzzard/
[01:10:16] Terence Tao
https://terrytao.wordpress.com/
book:
[00:08:19] The Proof in the Code
https://us.macmillan.com/books/9780374620059/theproofinthecode/
other:
[00:10:05] Brandolinis law
https://en.wikipedia.org/wiki/Brandolini%27s_law
[00:45:33] A new result on unit distances
https://openai.com/index/model-disproves-discrete-geometry-conjecture/
[01:01:08] Fermats Last Theorem formalisation
https://imperialcollegelondon.github.io/FLT/
paper:
[00:34:36] The Lean 4 theorem prover and programming language
https://doi.org/10.1007/978-3-030-79876-5_37
[01:02:06] AlphaProof
https://doi.org/10.1038/s41586-025-09833-y
[01:04:24] AlphaZero
https://arxiv.org/abs/1712.01815
RESCRIPT:
https://app.rescript.info/share/7d3d4a0059443236a01f6c9acbf4db58
https://app.rescript.info/api/public/sessions/b007264c0ce89047/pdf The Programming Language That Referees Mathematics — Leo de Moura](https://i.ytimg.com/vi/ZpQFebTK75A/mqdefault.jpg)


![Strange Geometric Shapes Found Inside AIs — Tom McGrath
Tom McGrath is co-founder and Chief Scientist at Goodfire, and a former Google DeepMind researcher. He joins Tim Scarfe to ask what neural networks actually learn, whether their internal representations converge on structures in the world, and whether interpretability can extract new scientific knowledge rather than merely explain model outputs.
Beginning with AlphaZero and learned modularity, the conversation moves into neural geometry: concept manifolds, reusable computation inside Llama, and why activation steering can fail when it pushes a model off-manifold. McGrath then makes the case for intentional design, using interpretability as part of the training loop. They examine controlled generalisation, features as rewards, predictive data debugging, and the uncomfortable fact that a model may recognise a hallucination or reward hack and still produce it.
The discussion closes on grader awareness, oversight and collusion between adaptive agents, then returns to sparse autoencoders. SAEs are useful, McGrath argues, but they may fracture the higher-dimensional structures networks actually use. This episode was made with support from Goodfire.
TIMESTAMPS:
00:00:00 Introduction: Can interpretability speed-run science?
00:02:03 The invisible grader
00:06:51 What AlphaZero learned from the world
00:12:24 Interpretability as a control loop
00:21:54 The forbidden method and safer interventions
00:37:36 Why models catch hallucinations too late
00:46:19 Debug the dataset before training
00:50:44 Why neural networks become modular
00:55:57 Finding the geometry inside a network
01:02:55 Why steering falls off the manifold
01:12:10 A reusable calculator inside Llama
01:17:19 From abstractions to goals
01:25:28 Reward hacking, oversight and collusion
01:37:23 Are sparse autoencoders dead?
REFERENCES:
paper:
[00:05:45] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
https://arxiv.org/abs/2502.17424v7
[00:11:05] Acquisition of Chess Knowledge in AlphaZero
https://arxiv.org/abs/2111.09259
[00:25:30] Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning
https://arxiv.org/abs/2507.16795
[00:29:30] Persona Vectors: Monitoring and Controlling Character Traits in Language Models
https://arxiv.org/abs/2507.21509
[00:41:14] Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability
https://arxiv.org/abs/2602.10067
[00:47:03] Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal
https://arxiv.org/abs/2606.12360
[01:00:26] Do Sparse Autoencoders Capture Concept Manifolds?
https://arxiv.org/abs/2604.28119
[01:03:04] Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
https://arxiv.org/abs/2605.05115
[01:14:20] Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
https://arxiv.org/abs/2605.01148
[01:29:35] Measuring Reward-Seeking via Contrastive Belief Updates
https://arxiv.org/abs/2607.18966v1
other:
[00:15:44] Intentional Design
https://www.goodfire.com/blog/intentional-design
[00:56:12] The World Inside Neural Networks
https://www.goodfire.com/research/the-world-inside-neural-networks
[01:37:28] A Pragmatic Vision for Interpretability
https://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability
RESCRIPT:
https://app.rescript.info/share/846cfee4131b664fd09209cc3b98018e Strange Geometric Shapes Found Inside AIs — Tom McGrath](https://i.ytimg.com/vi/_egu7OFem-k/mqdefault.jpg)
![AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]
Dr. Ilia Shumailov - Former DeepMind AI Security Researcher, now building security tools for AI agents
Ever wondered what happens when AI agents start talking to each other—or worse, when they start breaking things? Ilia Shumailov spent years at DeepMind thinking about exactly these problems, and hes here to explain why securing AI is way harder than you think.
**SPONSOR MESSAGES**
—
Check out notebooklm for your research project, its really powerful
https://notebooklm.google.com/
—
Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
Were racing toward a world where AI agents will handle our emails, manage our finances, and interact with sensitive data 24/7. But there is a problem. These agents are nothing like human employees. They never sleep, they can touch every endpoint in your system simultaneously, and they can generate sophisticated hacking tools in seconds. Traditional security measures designed for humans simply wont work.
Dr. Ilia Shumailov
https://x.com/iliaishacked
https://iliaishacked.github.io/
https://sequrity.ai/
TRANSCRIPT:
https://app.rescript.info/public/share/dVGsk8dz9_V0J7xMlwguByBq1HXRD6i4uC5z5r7EVGM
More from Ilia on our Patreon:
https://www.patreon.com/posts/116142401/ (interview from last year)
https://www.patreon.com/posts/ilia-shumailov-140359158 (extended version of this interview)
TOC:
00:00:00 - Introduction & Trusted Third Parties via ML
00:03:45 - Background & Career Journey
00:06:42 - Safety vs Security Distinction
00:09:45 - Prompt Injection & Model Capability
00:13:00 - Agents as Worst-Case Adversaries
00:15:45 - Personal AI & CAML System Defense
00:19:30 - Agents vs Humans: Threat Modeling
00:22:30 - Calculator Analogy & Agent Behavior
00:25:00 - IMO Math Solutions & Agent Thinking
00:28:15 - Diffusion of Responsibility & Insider Threats
00:31:00 - Open Source Security Concerns
00:34:45 - Supply Chain Attacks & Trust Issues
00:39:45 - Architectural Backdoors
00:44:00 - Academic Incentives & Defense Work
00:48:30 - Semantic Censorship & Halting Problem
00:52:00 - Model Collapse: Theory & Criticism
00:59:30 - Career Advice & Ross Anderson Tribute
REFS:
Lessons from Defending Gemini Against Indirect Prompt Injections
https://arxiv.org/abs/2505.14534
Defeating Prompt Injections by Design. Google, Google DeepMind, and ETH Zurich. (CAML)
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F.
https://arxiv.org/pdf/2503.18813
Agentic Misalignment: How LLMs could be insider threats
https://www.anthropic.com/research/agentic-misalignment
STOP ANTHROPOMORPHIZING INTERMEDIATE TOKENS AS REASONING/THINKING TRACES!
Subbarao Kambhampati et al
https://arxiv.org/pdf/2504.09762
Meiklejohn, S., Blauzvern, H., Maruseac, M., Schrock, S., Simon, L., & Shumailov, I. (2025).
Machine learning models have a supply chain problem.
https://arxiv.org/abs/2505.22778
Gao, Y., Shumailov, I., & Fawaz, K. (2025).
Supply-chain attacks in machine learning frameworks.
In Proceedings of the 8th MLSys Conference.
https://openreview.net/pdf?id=EH5PZW6aCr
Apache Log4j Vulnerability Guidance
https://www.cisa.gov/news-events/news/apache-log4j-vulnerability-guidance
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2023).
Architectural backdoors in neural networks.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 21163–21173).
Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2022).
Architectural backdoors in neural networks. arXiv preprint arXiv:2206.07840.
https://arxiv.org/pdf/2206.07840
Langford, H., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2024).
Architectural neural backdoors from first principles.
arXiv preprint arXiv:2402.06957.
Küchler, N., Petrov, I., Grobler, C., & Shumailov, I. (2025).
Architectural backdoors for within-batch data stealing and model inference manipulation.
arXiv preprint arXiv:2505.18323.
Position: Fundamental Limitations of LLM Censorship Necessitate New Approaches
David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Papernot, Vardan Papyan
https://proceedings.mlr.press/v235/glukhov24a.html
AlphaEvolve MLST interview [Matej Balog, Alexander Novikov]
https://www.youtube.com/watch?v=vC9nAosXrJw AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]](https://i.ytimg.com/vi/aoX_pGQMbEM/mqdefault.jpg)

![The Weird ChatGPT Hack That Leaked Training Data [Dr. Yannic Kilcher / Prof. Florian Tramer]
Dr. Yannic lightspeed Kilcher sits down with an AI Security Researcher Prof. Florian Tramer (an expert in the field, collaborating with folks like Nicholas Carlini and Milad Nasr at Google) to ruminate on the murky state of affairs in AI security.
Out of the gate they assert that AI detectors (those tools that try to spot if text is human-written or AI-generated) are basically doomed to fail—like trying to catch an eel with greasy hands blindfolded. Lightspeed Kilcher shares a funny story about how the word Delve kept rearing its ugly head in AI text because of training data emanating from Nigerian crowd workers, leading to false flags in job applications.
Tramer then takes the gloves off about security risks, saying that even a 99% success rate in AI means total failure in security terms—attackers will always find that 1% weak spot. They share an interesting example of a weird attack on ChatGPT: asking it to repeat poem forever made it spit out memorized internet scraps, leaking training data like a sieve. OpenAI patched it, but its like slapping a band-aid on a shark bite.
Lightspeed Kilcher asks about top worries, and Tramer lists three big ones in relatable terms: 1) Training AI on private data (like medical records) could lead to massive leaks, since models memorize stuff without us fully understanding how; 2) Prompt injections, where bad actors hijack AI agents to do harmful things, like Anthropics demo that controls computers—its like giving your keys to a stranger; and 3) The rush to deploy AI without safeguards, echoing past tech blunders like SQL hacks.
They reflect on how ChatGPT has lit a fire under AI security research—making it amazing and scary by turning hypotheticals into real-world issues with millions of users. The conversation touches on why just scaling up AI wont fix everything (we need causal understanding, not just more data), and why watermarking AI outputs isnt a silver bullet—its easy to bypass, especially with open-source models.
Its about time we had our old comrade light speed Kilcher back on MLST!
TRANSCRIPT:
https://app.rescript.info/public/share/PGieXFZCRyJjJswicatmNFia_pKCt_WVP-pZS4qguiw
LINKS:
Scalable Extraction of Training Data from (Production) Language Models [Poem Attack]
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel
https://arxiv.org/abs/2311.17035
- Prof. Florian Tramer
https://www.floriantramer.com/
- Yannic Kilchers YouTube channel
https://www.youtube.com/c/YannicKilcher
- Nicholas Carlinis research profile (collaborator on the data recovery project):
https://nicholas.carlini.com/ – Includes papers on AI attacks and memorization risks. The Weird ChatGPT Hack That Leaked Training Data [Dr. Yannic Kilcher / Prof. Florian Tramer]](https://i.ytimg.com/vi/c_hmxRVDXBE/mqdefault.jpg)
![AI training data will never be fully synthetic [SPONSORED]
We sat down with Sara Saab (VP of Product at Prolific) and Enzo Blindow (VP of Data and AI at Prolific) to explore the critical role of human evaluation in AI development and the challenges of aligning AI systems with human values. Prolific is a human annotation and orchestration platform for AI used by many of the major AI labs. This is a sponsored show in partnership with Prolific.
**SPONSOR MESSAGES**
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
While technologists want to remove humans from the loop for speed and efficiency, these non-deterministic AI systems actually require more human oversight than ever before. Prolifics approach is to put well-treated, verified, diversely demographic humans behind an API - making human feedback as accessible as any other infrastructure service.
When AI models like Grok 4 achieve top scores on technical benchmarks but feel awkward or problematic to use in practice, it exposes the limitations of our current evaluation methods. The guests argue that optimizing for benchmarks may actually weaken model performance in other crucial areas, like cultural sensitivity or natural conversation.
We also discuss Anthropics research showing that frontier AI models, when given goals and access to information, independently arrived at solutions involving blackmail - without any prompting toward unethical behavior. Even more concerning, the more sophisticated the model, the more susceptible it was to this agentic misalignment.
Enzo and Sarah present Prolifics Humane leaderboard as an alternative to existing benchmarking systems. By stratifying evaluations across diverse demographic groups, they reveal that different populations have vastly different experiences with the same AI models.
Looking forwards the guests imagine a world where humans take on coaching and teaching roles for AI systems - similar to how we might correct a child or review code. Working conditions and the evolution of labor in an AI-augmented world is also important to consider. Rather than replacing humans entirely, we may be moving toward more sophisticated forms of human-AI collaboration.
We need more representative evaluation frameworks that capture the messy reality of human values and cultural diversity.
Visit Prolific:
https://www.prolific.com/
Sara Saab (VP Product):
https://uk.linkedin.com/in/sarasaab
Enzo Blindow (VP Data & AI):
https://uk.linkedin.com/in/enzoblindow
TRANSCRIPT:
https://app.rescript.info/public/share/xZ31-0kJJ_xp4zFSC-bunC8-hJNkHpbm7Lg88RFcuLE
TOC:
[00:00:00] Intro & Background
[00:03:16] Human-in-the-Loop Challenges
[00:17:19] Can AIs Understand?
[00:32:02] Benchmarking & Vibes
[00:51:00] Agentic Misalignment Study
[01:03:00] Data Quality vs Quantity
[01:16:00] Future of AI Oversight
REFS:
Anthropic Agentic Misalignment
https://www.anthropic.com/research/agentic-misalignment
Value Compass
https://arxiv.org/pdf/2409.09586
Reasoning Models Don’t Always Say What They Think (Anthropic)
https://www.anthropic.com/research/reasoning-models-dont-say-think
https://assets.anthropic.com/m/71876fabef0f0ed4/original/reasoning_models_paper.pdf
Maslow’s Hierarchy Of Needs
https://www.simplypsychology.org/maslow.html
Apollo research - science of evals blog post
https://www.apolloresearch.ai/blog/we-need-a-science-of-evals
Leaderboard Illusion
https://www.youtube.com/watch?v=9W_OhS38rIE MLST video
The Leaderboard Illusion [2025]
Shivalika Singh, Yiyang Nan, Alex Wang, Daniel DSouza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, Sara Hooker
https://arxiv.org/abs/2504.20879
Humanities last exam
https://arxiv.org/abs/2501.14249
PRISM paper
https://arxiv.org/abs/2405.10254
https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback
Collective intelligence project
https://www.cip.org/whitepaper
https://ischool.utoronto.ca/faculty-profile/brian-cantwell-smith/
Ghost work (Mary Gray)
https://www.amazon.com/Ghost-Work-Silicon-Building-Underclass/dp/1328566242
Fairwork Cloudwork report
https://fair.work/en/ratings/cloudwork/
Gibson theory of affordances
https://cs.brown.edu/courses/cs137/2017/readings/Gibson-AFF.pdf AI training data will never be fully synthetic [SPONSORED]](https://i.ytimg.com/vi/cnxZZTl1tkk/mqdefault.jpg)

![A Physicist Found the Hidden Phase Transitions in Society — Cristopher Moore
We are joined by Cristopher Moore, a professor at the Santa Fe Institute with a diverse background in physics, computer science, and machine learning.
The conversation begins with Cristopher, who calls himself a frog explaining that he prefers to dive deep into specific, concrete problems rather than taking a high-level birds-eye view.
They explore why current AI models, like transformers, are so surprisingly effective. Cristopher argues its because the real world isnt random; its full of rich structures, patterns, and hierarchies that these models can learn to exploit, even if we dont fully understand how.
ADDITIONAL FREE CONTENT:
We also filmed Cris talk at DISI, the audio quality was too bad for us to officially publish, slides included
https://www.dropbox.com/scl/fo/bn0tn7z9fxrh1rya1ycrt/AH_d4SOOvlauxLzBgfnNUA0?rlkey=ec65ero4feo5uezum0j6p9z4t&dl=0
**SPONSORS**
Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy (note we got url wrong in the video, this is correct URL!)
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
***
Cristopher Moore:
https://sites.santafe.edu/~moore/
TOC:
00:00:00 - Introduction
00:02:05 - Meet Christopher Moore: A Frog in the World of Science
00:05:14 - The Limits of Transformers and Real-World Data
00:11:19 - Intelligence as Creative Problem-Solving
00:23:30 - Grounding, Meaning, and Shared Reality
00:31:09 - The Nature of Creativity and Aesthetics
00:44:31 - Computational Irreducibility and Universality
00:53:06 - Turing Completeness, Recursion, and Intelligence
01:11:26 - The Universe Through a Computational Lens
01:26:45 - Algorithmic Justice and the Need for Transparency
TRANSCRIPT: https://app.rescript.info/public/share/VRe2uQSvKZOm0oIBoDsrNwt46OMCqRnShVnUF3qyoFk
Filmed at DISI (Diverse Intelligences Summer Institute)
https://disi.org/
REFS:
The Nature of computation [Chris Moore]
https://nature-of-computation.org/
Birds and Frogs [Freeman Dyson]
https://www.ams.org/notices/200902/rtx090200212p.pdf
Replica Theory [Parisi et al]
https://arxiv.org/pdf/1409.2722
Janossy pooling [Fabian Fuchs]
https://fabianfuchsml.github.io/equilibriumaggregation/
Cracking the cryptic [YT channel]
https://www.youtube.com/c/CrackingTheCryptic
Sudoko Bench [Sakana]
https://sakana.ai/sudoku-bench/
Fractured entangled representations “phylogenetic locking in comment” [Kumar/Stanley]
https://arxiv.org/pdf/2505.11581 (see our shows on this)
The War Against Cliché: [Martin Amis]
https://www.amazon.com/War-Against-Cliche-Reviews-1971-2000/dp/0375727167
Rule 110 (CA)
https://mathworld.wolfram.com/Rule150.html
Universality in Elementary Cellular Automata [Matt Cooke]
https://wpmedia.wolfram.com/sites/13/2018/02/15-1-1.pdf
Small Semi-Weakly Universal Turing Machines [Damien Woods]
https://tilde.ini.uzh.ch/users/tneary/public_html/WoodsNeary-FI09.pdf
COMPUTING MACHINERY AND INTELLIGENCE [Turing, 1950]
https://courses.cs.umbc.edu/471/papers/turing.pdf
Comment on Space Time as a causal set [Moore, 88]
https://sites.santafe.edu/~moore/comment.pdf
Recursion Theory on the Reals and Continuous-time Computation [Moore, 96] A Physicist Found the Hidden Phase Transitions in Society — Cristopher Moore](https://i.ytimg.com/vi/dB9lJkUkIUM/mqdefault.jpg)