Uploaded October 2021 | Updated November 2026, 3 hours ago
Trenton Bricken, Harvard University
Abstract: While Attention has come to be an important mechanism in deep learning, it emerged out of a heuristic process of trial and error, providing limited intuition for why it works so well. Here, we show that Transformer Attention closely approximates Sparse Distributed Memory (SDM), a biologically plausible associative memory model, under certain data conditions. We confirm that these conditions are satisfied in pre-trained GPT2 Transformer models. We discuss the implications of the Attention-SDM map and provide new computational and biological interpretations of Attention.
Trenton Bricken, Harvard University
Abstract: While Attention has come to be an important mechanism in deep learning, it emerged out of a heuristic process of trial and error, providing limited intuition for why it works so well. Here, we show that Transformer Attention closely approximates Sparse Distributed Memory (SDM), a biologically plausible associative memory model, under certain data conditions. We confirm that these conditions are satisfied in pre-trained GPT2 Transformer models. We discuss the implications of the Attention-SDM map and provide new computational and biological interpretations of Attention.




![A Conversation with Karl Friston | The Intelligence Question
What is intelligence? How does it work, and are there unified principles that could explain brains, minds, and machines? In this conversation, recorded at University College London, CBMMs John Gabrieli Buchet talks with Karl Friston, the neuroscientist behind the free energy principle and active inference, about what the brain is doing when it perceives, acts, and learns; where todays AI systems fall short; and what the path forward might look like toward more human-like systems.
This is the first episode of The Intelligence Question, a new CBMM/Poggio Lab series consisting of in-depth conversations with leading researchers across neuroscience, AI, and related fields about the foundational principles of intelligence. Companion transcript and future episodes: [LINK]
Karl Friston is a professor of neuroscience at University College London and the Scientific Director of the Wellcome Centre for Human Neuroimaging. He is best known for his work on statistical parametric mapping, dynamic causal modeling, and the free energy principle.
Read the full blog post here - https://poggio-lab.mit.edu/blogsupdates A Conversation with Karl Friston | The Intelligence Question](https://i.ytimg.com/vi/WWC31kkYjF0/mqdefault.jpg)





