Recycling Fine-Tuned Models to Pretrain – Leshem Choshen @HopkinsEngineer
Recycling Fine-Tuned Models to Pretrain – Leshem Choshen  @HopkinsEngineer
Uploaded October 2024 | Updated November 2026, 10 hours ago
Computer Science Seminar Series
October 17, 2024

“Recycling Fine-Tuned Models to Pretrain (on Loss Spaces, Fusing, and Evolving Pretraining)”
Leshem Choshen, Massachusetts Institute of Technology & IBM

Prohibitive pretraining costs makes pretraining research a rare sight—however, this is not the case for analyzing, using, and fine-tuning those models. This talk focuses on one option to improve models in a scientific way, in small measurable steps; specifically, it introduces the concept of merging multiple fine-tuned/parameter-efficient fine-tuning models into one and discusses works tackling what we understand about it, how it works, more up-to-date methods, and how iteratively merging models may allow collaborative continual pretraining.

Leshem Choshen is a postdoctoral researcher at the Massachusetts Institute of Technology and IBM who aims to study model development openly and collaboratively, allow feasible pretraining research, and evaluate efficiently. To do so, they co-created model merging, TIES merging, the BabyLM Challenge. They were chosen for postdoctoral Rothschild and Fulbright fellowships and received a Best PhD Thesis Award from the Israeli Association for Artificial Intelligence, as well as a Blavatnik Prize for Computer Science. With broad natural language processing and machine learning interests, Choshen has also worked on reinforcement learning, understanding how neural networks learn, and Project Debater, the first machine system capable of holding a formal debate (as of 2019), which was featured on the cover of Nature.
Recycling Fine-Tuned Models to Pretrain – Leshem ChoshenDesigning Machine Learning Processes for Equitable Health Systems – Marzyeh GhassemiBuilding Haystacksto Find Needles – Erik RyeMechanical Engineering Freshman Design Exhibition 2026Good Research Karma: The Unexpected Benefits of Striving for Algorithmic Fairness – Omer ReingoldCreating Robustness and Resiliency in Distributed Autonomous Groups of Robots or DronesAdversarial Robustness and Forensics for Deep Neural Networks – Ben Y. ZhaoMachine Learning & Causality: Building Efficient, Reliable Models for Decision-Making – Maggie MakarThe 2026 Charles and Mary O’Melia LectureData-Distributional Approaches for Generalizable Language Models – Sang Michael XieThe Installation of Jordan J. Green as the Herschel L. Seder ProfessorAI: Accelerating Materials Development
Johns Hopkins Engineering |

Recycling Fine-Tuned Models to Pretrain – Leshem Choshen

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER