Why Pretrained Models Hallucinate @SimonsInstitute
Why Pretrained Models Hallucinate  @SimonsInstitute
Uploaded October 2026 | Updated November 2026, 4 days ago
Santosh Vempala (Georgia Institute of Technology)
https://simons.berkeley.edu/talks/santosh-vempala-georgia-institute-technology-2026-10-07
Trustworthy AI: From Hallucinations to Reliable Autonomy

We analyze hallucination in language models from a mathematical perspective and find that the statistical pressures of the next-word prediction training pipeline induce hallucinations under fairly general conditions even when trained on clean data. The core proof is a simple reduction from a natural classification task ("fact or hallucination") to a generation/prompt completion task.

The talk is based on facts and on work with Adam T. Kalai (STOC 2025) and with Kalai, Ofer Nachum and Eddie Zhang (Nature 2026).
Why Pretrained Models HallucinatePeaked quantum advantage using error correctionThe elusive generalization: classical bounds to double descent to grokkingMatrix Concentration and Equivalent Operators on Fock SpacesRandomness Extractors and Related Objects (Part 2)Sampling from spherical spin glasses: diffusions and simulated annealingTalk by Max Hopkins (Institute for Advanced Study)Sequential Statistical Methods for Reliable Machine LearningNodal surplus and the spectral decomposition of the cycle spaceSturmian discrete and metric graphs - appearance and absence of gapsPseudorandomness against AM circuits and new oracle separationsWhat Really Separates Autoregression and Diffusion? A Synthesis and Path Beyond
Simons Institute for the Theory of Computing |

Why Pretrained Models Hallucinate

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER