Uploaded April 2026 | Updated November 2026, 1 hour ago
The behavior of large language models---like those underlying ChatGPT---is not programmed; it is the emergent result of centuries of computation and an internet's worth of text. Understanding what goes on inside these language models is both one of the most exciting scientific opportunities, and one of the most pressing engineering challenges of our time. In this work, I'll provide an overview of this problem, and then discuss work my colleagues and I have done to discover what goes on inside these systems: do they learn the structure of human languages? How can we discover what concepts or ideas they have arrived at, which humans might not have thought of yet? And how can we make precision fixes for their problems?
The behavior of large language models---like those underlying ChatGPT---is not programmed; it is the emergent result of centuries of computation and an internet's worth of text. Understanding what goes on inside these language models is both one of the most exciting scientific opportunities, and one of the most pressing engineering challenges of our time. In this work, I'll provide an overview of this problem, and then discuss work my colleagues and I have done to discover what goes on inside these systems: do they learn the structure of human languages? How can we discover what concepts or ideas they have arrived at, which humans might not have thought of yet? And how can we make precision fixes for their problems?










