Data-Distributional Approaches for Generalizable Language Models – Sang Michael Xie @HopkinsEngineer
Data-Distributional Approaches for Generalizable Language Models – Sang Michael Xie  @HopkinsEngineer
Uploaded August 2024 | Updated November 2026, 4 days ago
Computer Science Seminar Series
March 25, 2024

“Data-Distributional Approaches for Generalizable Language Models”
Sang Michael Xie, Stanford University

High-quality datasets are crucial for improving the capabilities and training efficiency of large language models. However, current datasets are typically prepared in an ad hoc, heuristic way. In this talk, Sang Michael Xie will present principled approaches to improving and understanding language models centered on the pre-training data distribution. First, he will describe how to improve the efficiency of training multipurpose language models by optimizing the mixture of data sources with robust optimization. Second, he will discuss an efficient importance resampling method for selecting relevant data from trillion-token-scale web datasets for training a specialized model. Finally, he will introduce a first theoretical analysis of in-context learning, a key capability of language models to learn from examples in a textual prompt, that traces the capability back to modeling coherence structure in the pre-training data.

Sang Michael Xie is a computer science PhD student at Stanford University advised by Percy Liang and Tengyu Ma. His research focuses on data-centric machine learning for language models, understanding pre-training and adaptation, and pre-training and self-training methods for robust machine learning. Xie was awarded a NDSEG Fellowship and was previously a student researcher at Google Brain. His work has been recognized as one of Scientific American's World-Changing Ideas, published in flagship venues such as Science, and covered by media outlets including The New York Times, The Washington Post, Reuters, BBC News, IEEE Spectrum, and The Verge.
Data-Distributional Approaches for Generalizable Language Models – Sang Michael XieThe Installation of Jordan J. Green as the Herschel L. Seder ProfessorAI: Accelerating Materials DevelopmentJohns Hopkins Engineering Live StreamKnowing What AI Systems Don’t Know and Why it Matters – Jim HendlerLearning and Control for Safety, Efficiency, and Resiliency of Embodied AI – Fei MiaoDesigning Cooperative and Socially-Aware Autonomy – Alyssa PiersonBeyond the Benchmarking Paradigm – Inioluwa Deborah RajiISPEED Summer Program in Biomedical Engineering at Johns HopkinsThe Installation of Paulette Clancy as the Edward J. Schaefer ProfessorVirtual Tour of Latrobe HallThe Network as a Programmable Platform – Larry Peterson
Johns Hopkins Engineering |

Data-Distributional Approaches for Generalizable Language Models – Sang Michael Xie

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER