AI Interpretability, Safety, and Meaning - Nora Belrose @MachineLearningStreetTalk

Uploaded November 2024 | Updated September 2026, 1 week ago

SPONSOR MESSAGES:
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.
centml.ai/pricing

Nora Belrose, Head of Interpretability Research at EleutherAI, delivers a wide-ranging conversation that moves from the mathematical foundations of concept erasure in neural networks to fundamental questions about consciousness, AI safety, and Buddhist philosophy.

The technical core centers on LEACE (LEAst-squares Concept Erasure), a method Belrose developed for surgically removing targeted information from neural network representations. She explains how LEACE emerged from connecting two prior approaches (RLACE and spectral attribute removal) through a mathematical equivalence proof, and demonstrates its applications in both fairness-oriented debiasing and interpretability research. A key finding: language models remain functional even after erasing part-of-speech information from every layer, suggesting robust reliance on redundant cues.

Belrose then presents her ICML paper on simplicity biases in deep learning, showing that neural networks learn to exploit statistical moments in order -- first means, then covariances, then higher-order statistics. This has implications for understanding when and why concept erasure techniques may backfire against sufficiently deep models.

The second half pivots to AI safety, where Belrose delivers a detailed critique of "counting arguments" used to predict AI misalignment. She argues these arguments rely on the principle of indifference applied to poorly-defined outcome spaces, drawing an analogy to an identical argument structure that would absurdly predict all neural networks must overfit. She connects this to broader questions about goal attribution, agency, and whether instrumental convergence arguments hold up under scrutiny.

The conversation concludes with an exploration of 4E cognition, Evan Thompson's philosophy of mind, Belrose's departure from effective altruism, and her growing interest in Buddhist philosophy as a framework for thinking about meaning in a post-automation world.

---
REFERENCES:
Paper:
[00:00:00] Episode Shownotes
dropbox.com/scl/fi/38fhsv2zh8gnubtjaoq4a/NORA_FINAL.pdf?rlkey=0e5r8rd261821g1em4dgv0k70&st=t5c9ckfb&dl=0
[00:05:00] LEACE Paper
arxiv.org/abs/2306.03819
[00:06:40] RLACE Paper
arxiv.org/abs/2201.12091
[00:08:20] Spectral Attribute Removal
arxiv.org/abs/2012.14424
[00:15:00] Pythia Models
arxiv.org/abs/2304.01373
[00:20:30] LoRA
arxiv.org/abs/2106.09685
[02:00:00] Holden Karnofsky
forum.effectivealtruism.org/posts/T975ydo3mx4YnRv4J/ea-is-about-maximization-and-maximization-is-perilous
Company:
[00:01:37] CentML
centml.ai/pricing
[00:01:37] Tufa AI Labs
tufalabs.ai
[00:02:20] EleutherAI
eleuther.ai
Person:
[00:02:20] Nora Belrose
norabelrose.com
[01:03:00] Evan Thompson
evanthompson.me

---
LINKS:
Full Transcript: app.rescript.info/share/79d69cf24406cc36d8f7e8eee389e3ae
Download PDF transcript: app.rescript.info/api/public/sessions/61e64e1737593802/pdf

Nora Belrose:
norabelrose.com
scholar.google.com/citations?user=p_oBc64AAAAJ&hl=en
https://x.com/norabelrose

AI Interpretability, Safety, and Meaning - Nora Belrose

This is what happens when you let AIs debate

Microsofts professor Chris Bishop on the Sparks of AGI

AI Doesnt Need Intelligence. Thats the Whole Point. - Luciano Floridi

NEURAL NETWORKS ARE WEIRD! - Neel Nanda (DeepMind)

Biologically-inspired AI and Mortal Computation

A CERN like effort for AI (Jakob Foerster)

$Strange Geometric Shapes Found Inside AIs — Tom McGrath Tom McGrath is co-founder and Chief Scientist at Goodfire, and a former Google DeepMind researcher. He joins Tim Scarfe to ask what neural networks actually learn, whether their internal representations converge on structures in the world, and whether interpretability can extract new scientific knowledge rather than merely explain model outputs. Beginning with AlphaZero and learned modularity, the conversation moves into neural geometry: concept manifolds, reusable computation inside Llama, and why activation steering can fail when it pushes a model off-manifold. McGrath then makes the case for intentional design, using interpretability as part of the training loop. They examine controlled generalisation, features as rewards, predictive data debugging, and the uncomfortable fact that a model may recognise a hallucination or reward hack and still produce it. The discussion closes on grader awareness, oversight and collusion between adaptive agents, then returns to sparse autoencoders. SAEs are useful, McGrath argues, but they may fracture the higher-dimensional structures networks actually use. This episode was made with support from Goodfire. TIMESTAMPS: 00:00:00 Introduction: Can interpretability speed-run science? 00:02:03 The invisible grader 00:06:51 What AlphaZero learned from the world 00:12:24 Interpretability as a control loop 00:21:54 The forbidden method and safer interventions 00:37:36 Why models catch hallucinations too late 00:46:19 Debug the dataset before training 00:50:44 Why neural networks become modular 00:55:57 Finding the geometry inside a network 01:02:55 Why steering falls off the manifold 01:12:10 A reusable calculator inside Llama 01:17:19 From abstractions to goals 01:25:28 Reward hacking, oversight and collusion 01:37:23 Are sparse autoencoders dead? REFERENCES: paper: [00:05:45] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs https://arxiv.org/abs/2502.17424v7 [00:11:05] Acquisition of Chess Knowledge in AlphaZero https://arxiv.org/abs/2111.09259 [00:25:30] Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning https://arxiv.org/abs/2507.16795 [00:29:30] Persona Vectors: Monitoring and Controlling Character Traits in Language Models https://arxiv.org/abs/2507.21509 [00:41:14] Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability https://arxiv.org/abs/2602.10067 [00:47:03] Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal https://arxiv.org/abs/2606.12360 [01:00:26] Do Sparse Autoencoders Capture Concept Manifolds? https://arxiv.org/abs/2604.28119 [01:03:04] Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior https://arxiv.org/abs/2605.05115 [01:14:20] Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts https://arxiv.org/abs/2605.01148 [01:29:35] Measuring Reward-Seeking via Contrastive Belief Updates https://arxiv.org/abs/2607.18966v1 other: [00:15:44] Intentional Design https://www.goodfire.com/blog/intentional-design [00:56:12] The World Inside Neural Networks https://www.goodfire.com/research/the-world-inside-neural-networks [01:37:28] A Pragmatic Vision for Interpretability https://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability RESCRIPT: https://app.rescript.info/share/846cfee4131b664fd09209cc3b98018e$