Uploaded January 2025 | Updated September 2026, 8 hours ago
Abstract:
Have foundation models fallen short of their promise? Practitioners are eager to adapt these models to specialized use cases, yet doing so remains a major pain point. Foundation models, like large language models, go through multiple stages of training, such as pre-training, followed by supervised fine-tuning and human preference learning. This makes it harder for them to adapt to newer tasks, as updating their parameters can undo the benefits of the earlier training stages. In this talk, I will describe data-centric
approaches to easily adapt foundation models to new tasks.
In the first part, I introduce Bonito, an open-source large language model that converts users' unannotated data from specialized domains into synthetic instruction tuning datasets. Collecting instructions from experts is the primary bottleneck in building specialized language models, as it is both expensive and time-consuming. With Bonito, practitioners can rapidly create synthetic instructions to train specialized language models without human annotations.
In the second part, I introduce compositional soft prompting, a parameter-efficient learning method for adapting vision-language models that learns to identify objects by composing concepts. This method enables the model to generalize to new, unseen compositions without annotations. This compositionality framework allows us to systematically study fundamental properties, such as concept binding, in vision-language models and understand their limitations.
I will conclude the talk by proposing ways to improve intelligent systems for scientific discovery by creating better foundation models for understanding scientific literature and a cost-effective expert-in-the-loop evaluation framework.
Bio:
Nihal Nayak is a sixth-year Ph.D. student in the Department of Computer Science at Brown University, where he is advised by Stephen Bach. His research focuses on building zero-shot systems that generalize to new tasks without human annotations. His work has been published in leading machine learning conferences and journals, including ICLR, ACL and EACL Findings, JMLR, TMLR, ACL Demo, and MLSys. He is the recipient of the Andries van Dam Graduate Fellowship.
Abstract:
Have foundation models fallen short of their promise? Practitioners are eager to adapt these models to specialized use cases, yet doing so remains a major pain point. Foundation models, like large language models, go through multiple stages of training, such as pre-training, followed by supervised fine-tuning and human preference learning. This makes it harder for them to adapt to newer tasks, as updating their parameters can undo the benefits of the earlier training stages. In this talk, I will describe data-centric
approaches to easily adapt foundation models to new tasks.
In the first part, I introduce Bonito, an open-source large language model that converts users' unannotated data from specialized domains into synthetic instruction tuning datasets. Collecting instructions from experts is the primary bottleneck in building specialized language models, as it is both expensive and time-consuming. With Bonito, practitioners can rapidly create synthetic instructions to train specialized language models without human annotations.
In the second part, I introduce compositional soft prompting, a parameter-efficient learning method for adapting vision-language models that learns to identify objects by composing concepts. This method enables the model to generalize to new, unseen compositions without annotations. This compositionality framework allows us to systematically study fundamental properties, such as concept binding, in vision-language models and understand their limitations.
I will conclude the talk by proposing ways to improve intelligent systems for scientific discovery by creating better foundation models for understanding scientific literature and a cost-effective expert-in-the-loop evaluation framework.
Bio:
Nihal Nayak is a sixth-year Ph.D. student in the Department of Computer Science at Brown University, where he is advised by Stephen Bach. His research focuses on building zero-shot systems that generalize to new tasks without human annotations. His work has been published in leading machine learning conferences and journals, including ICLR, ACL and EACL Findings, JMLR, TMLR, ACL Demo, and MLSys. He is the recipient of the Andries van Dam Graduate Fellowship.
