NEURAL NETWORKS ARE WEIRD! - Neel Nanda (DeepMind) @MachineLearningStreetTalk

Uploaded December 2024 | Updated September 2026, 1 week ago

SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.
centml.ai/pricing

Neel Nanda leads the mechanistic interpretability team at Google DeepMind. At 26, he's become one of the most prominent researchers working on the question of what's actually going on inside neural networks -- systems that can win IMO medals and write complex software, but which nobody actually designed or understands.

This nearly four-hour conversation is a deep technical dive into the field. Nanda explains why machine learning is fundamentally weird: we produce artifacts that do impressive things, but unlike conventional software, no one wrote the code or planned the architecture. His team's goal is reverse-engineering these systems by finding the internal structures and algorithms that emerge during training.

The discussion covers the mechanics of sparse autoencoders at length -- how they decompose model activations into interpretable feature vectors, the mathematical foundations (ReLU vs TopK activation functions), scaling laws for feature learning, and the engineering challenges of running them at the scale of frontier models. Nanda walks through the Golden Gate Claude experiment (amplifying a single feature to make Claude obsessed with the Golden Gate Bridge), induction heads (the circuits responsible for in-context learning), and activation patching as a causal intervention technique.

On AI safety, Nanda is pragmatic. He argues that mechanistic interpretability gives us genuine empirical evidence about questions that are otherwise stuck in philosophical debate -- do models have goals? Do they deceive? He also discusses the limitations: sparse autoencoders haven't yet demonstrated capabilities beyond what fine-tuning already achieves, and at sufficient model complexity, models could potentially facade interpretability measurements. The conversation covers his path from pure maths at Cambridge through Anthropic to DeepMind, and why he thinks hands-on coding matters more than reading papers for new researchers entering the field.

---
REFERENCES:
person:
[00:00:00] Neel Nanda - Personal Website
neelnanda.io
tool:
[00:35:00] TransformerLens
github.com/TransformerLensOrg/TransformerLens
paper:
[01:00:31] A Mathematical Framework for Transformer Circuits
https://transformer-circuits.pub/2021/framework/index.html
[01:01:40] In-context Learning and Induction Heads
https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html
[01:21:06] Scaling Monosemanticity
https://transformer-circuits.pub/2024/scaling-monosemanticity/
[01:33:27] Refusal in Language Models Is Mediated by a Single Direction
arxiv.org/abs/2406.11717

---
LINKS:
Full Transcript: app.rescript.info/share/acb415fa59ae2d2909d60d761c8f4ff4
Download PDF transcript: app.rescript.info/api/public/sessions/c3a4bf1e32a46ce7/pdf

NEEL NANDA:
neelnanda.io
scholar.google.com/citations?user=GLnX3MkAAAAJ&hl=en
https://x.com/NeelNanda5

NEURAL NETWORKS ARE WEIRD! - Neel Nanda (DeepMind)

Biologically-inspired AI and Mortal Computation

A CERN like effort for AI (Jakob Foerster)

$Strange Geometric Shapes Found Inside AIs — Tom McGrath Tom McGrath is co-founder and Chief Scientist at Goodfire, and a former Google DeepMind researcher. He joins Tim Scarfe to ask what neural networks actually learn, whether their internal representations converge on structures in the world, and whether interpretability can extract new scientific knowledge rather than merely explain model outputs. Beginning with AlphaZero and learned modularity, the conversation moves into neural geometry: concept manifolds, reusable computation inside Llama, and why activation steering can fail when it pushes a model off-manifold. McGrath then makes the case for intentional design, using interpretability as part of the training loop. They examine controlled generalisation, features as rewards, predictive data debugging, and the uncomfortable fact that a model may recognise a hallucination or reward hack and still produce it. The discussion closes on grader awareness, oversight and collusion between adaptive agents, then returns to sparse autoencoders. SAEs are useful, McGrath argues, but they may fracture the higher-dimensional structures networks actually use. This episode was made with support from Goodfire. TIMESTAMPS: 00:00:00 Introduction: Can interpretability speed-run science? 00:02:03 The invisible grader 00:06:51 What AlphaZero learned from the world 00:12:24 Interpretability as a control loop 00:21:54 The forbidden method and safer interventions 00:37:36 Why models catch hallucinations too late 00:46:19 Debug the dataset before training 00:50:44 Why neural networks become modular 00:55:57 Finding the geometry inside a network 01:02:55 Why steering falls off the manifold 01:12:10 A reusable calculator inside Llama 01:17:19 From abstractions to goals 01:25:28 Reward hacking, oversight and collusion 01:37:23 Are sparse autoencoders dead? REFERENCES: paper: [00:05:45] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs https://arxiv.org/abs/2502.17424v7 [00:11:05] Acquisition of Chess Knowledge in AlphaZero https://arxiv.org/abs/2111.09259 [00:25:30] Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning https://arxiv.org/abs/2507.16795 [00:29:30] Persona Vectors: Monitoring and Controlling Character Traits in Language Models https://arxiv.org/abs/2507.21509 [00:41:14] Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability https://arxiv.org/abs/2602.10067 [00:47:03] Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal https://arxiv.org/abs/2606.12360 [01:00:26] Do Sparse Autoencoders Capture Concept Manifolds? https://arxiv.org/abs/2604.28119 [01:03:04] Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior https://arxiv.org/abs/2605.05115 [01:14:20] Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts https://arxiv.org/abs/2605.01148 [01:29:35] Measuring Reward-Seeking via Contrastive Belief Updates https://arxiv.org/abs/2607.18966v1 other: [00:15:44] Intentional Design https://www.goodfire.com/blog/intentional-design [00:56:12] The World Inside Neural Networks https://www.goodfire.com/research/the-world-inside-neural-networks [01:37:28] A Pragmatic Vision for Interpretability https://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability RESCRIPT: https://app.rescript.info/share/846cfee4131b664fd09209cc3b98018e$

AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]

Is the Mind More Than Just the Brain? - Tom Froese

The Weird ChatGPT Hack That Leaked Training Data [Dr. Yannic Kilcher / Prof. Florian Tramer]

AI training data will never be fully synthetic [SPONSORED]

A Physicist Found the Hidden Phase Transitions in Society — Cristopher Moore

Why US AI Act Compute Thresholds Are Misguided...