Uploaded November 2023 | Updated September 2026, 3 days ago
Abstract:
There is growing evidence that pretraining on high quality, carefully thought-out tokens such as code or mathematics plays an important role in improving the reasoning abilities of large language models. For example, Minerva, a PaLM model finetuned on billions of tokens of mathematical documents from arXiv and the web, reported dramatically improved performance on problems that require quantitative reasoning. However, because all known open source web datasets employ preprocessing that does not faithfully preserve mathematical notation, the benefits of large scale training on quantitive web documents are unavailable to the research community. We introduce OpenWebMath, an open dataset inspired by these works containing 14.7B tokens of mathematical webpages from Common Crawl. We describe in detail our method for extracting text and LaTeX content and removing boilerplate from HTML documents, as well as our methods for quality filtering and deduplication. Additionally, we run small-scale experiments by training 1.4B parameter language models on OpenWebMath, showing that models trained on 14.7B tokens of our dataset surpass the performance of models trained on over 20x the amount of general language data. We hope that our dataset, openly released on the Hugging Face Hub, will help spur advances in the reasoning abilities of large language models.
Bio:
Keiran Paster is a fifth year PhD student supervised by Jimmy Ba and Sheila McIlraith at the University of Toronto. Keiran’s research lies in the intersection of sequence modeling and decision-making. His work has formalized how sequence models can be made to make sequential decisions, even in stochastic environments through specially optimized prompts. This work lead to the creation of a groundbreaking agent called STEVE-1 that can play Minecraft from pixels with keyboard and mouse actions while following both text and visual instructions through the fine-tuning of foundation models. Keiran also works on LLM reasoning, including work on automatic prompt engineering and its application for optimizing prompts for performant and general reasoning, as well as work on a web-scale math dataset and corresponding powerful foundation model for reasoning. Keiran also worked on data efficiency for frontier models within Google Research.
Abstract:
There is growing evidence that pretraining on high quality, carefully thought-out tokens such as code or mathematics plays an important role in improving the reasoning abilities of large language models. For example, Minerva, a PaLM model finetuned on billions of tokens of mathematical documents from arXiv and the web, reported dramatically improved performance on problems that require quantitative reasoning. However, because all known open source web datasets employ preprocessing that does not faithfully preserve mathematical notation, the benefits of large scale training on quantitive web documents are unavailable to the research community. We introduce OpenWebMath, an open dataset inspired by these works containing 14.7B tokens of mathematical webpages from Common Crawl. We describe in detail our method for extracting text and LaTeX content and removing boilerplate from HTML documents, as well as our methods for quality filtering and deduplication. Additionally, we run small-scale experiments by training 1.4B parameter language models on OpenWebMath, showing that models trained on 14.7B tokens of our dataset surpass the performance of models trained on over 20x the amount of general language data. We hope that our dataset, openly released on the Hugging Face Hub, will help spur advances in the reasoning abilities of large language models.
Bio:
Keiran Paster is a fifth year PhD student supervised by Jimmy Ba and Sheila McIlraith at the University of Toronto. Keiran’s research lies in the intersection of sequence modeling and decision-making. His work has formalized how sequence models can be made to make sequential decisions, even in stochastic environments through specially optimized prompts. This work lead to the creation of a groundbreaking agent called STEVE-1 that can play Minecraft from pixels with keyboard and mouse actions while following both text and visual instructions through the fine-tuning of foundation models. Keiran also works on LLM reasoning, including work on automatic prompt engineering and its application for optimizing prompts for performant and general reasoning, as well as work on a web-scale math dataset and corresponding powerful foundation model for reasoning. Keiran also worked on data efficiency for frontier models within Google Research.
