Uploaded October 2025 | Updated September 2026, 1 day ago
As large language models (LLMs) become integral to scientific research, a critical challenge emerges: their knowledge of science rapidly becomes outdated. While recent methods aim to refresh or augment LLMs with new findings, we argue that ideal scientific knowledge updates require more than simply adding information—they must preserve existing understanding, incorporate new discoveries, and enable reasoning about the emerging future. In this talk, Yike Wang introduces ScienceMeter, a new framework for evaluating how well LLMs update their scientific knowledge across three dimensions: preservation, acquisition, and projection. ScienceMeter operationalizes scientific knowledge as atomic scientific claims, and evaluates using judgment and generation tasks across a curated dataset of over 15,000 papers and 30,000 claims spanning ten scientific domains. The study with five representative update methods, across training- and inference-time, shows that the best-performing methods achieve only 85.9% preservation, 71.7% acquisition, and 37.7% (or more) projection. Inference-time updates work for large models, whereas smaller models require training-based methods. No method achieves robust performance across all dimensions and domains, highlighting that developing reliable and helpful scientific knowledge update mechanisms for LLMs remains an open and crucial challenge.
Yike Wang is a Ph.D. student at the University of Washington, advised by Professor Hanna Hajishirzi and Professor Yulia Tsvetkov. Her research focuses on developing reliable and helpful large language models, particularly in the domain of science. She holds bachelor’s and master’s degrees in computer science and mathematics from UC Berkeley. Her website is yikee.github.io/.
As large language models (LLMs) become integral to scientific research, a critical challenge emerges: their knowledge of science rapidly becomes outdated. While recent methods aim to refresh or augment LLMs with new findings, we argue that ideal scientific knowledge updates require more than simply adding information—they must preserve existing understanding, incorporate new discoveries, and enable reasoning about the emerging future. In this talk, Yike Wang introduces ScienceMeter, a new framework for evaluating how well LLMs update their scientific knowledge across three dimensions: preservation, acquisition, and projection. ScienceMeter operationalizes scientific knowledge as atomic scientific claims, and evaluates using judgment and generation tasks across a curated dataset of over 15,000 papers and 30,000 claims spanning ten scientific domains. The study with five representative update methods, across training- and inference-time, shows that the best-performing methods achieve only 85.9% preservation, 71.7% acquisition, and 37.7% (or more) projection. Inference-time updates work for large models, whereas smaller models require training-based methods. No method achieves robust performance across all dimensions and domains, highlighting that developing reliable and helpful scientific knowledge update mechanisms for LLMs remains an open and crucial challenge.
Yike Wang is a Ph.D. student at the University of Washington, advised by Professor Hanna Hajishirzi and Professor Yulia Tsvetkov. Her research focuses on developing reliable and helpful large language models, particularly in the domain of science. She holds bachelor’s and master’s degrees in computer science and mathematics from UC Berkeley. Her website is yikee.github.io/.
