Uploaded June 2023 | Updated September 2026, 1 day ago
Abstract: We are approaching a future where text generation technologies will allow us to generate texts that are not only fluent at a surface level, but also coherent in their overall structure. To enable this future, my research
focuses on evaluating and improving structure modeling in language models. In the first part of this talk, I will introduce a method for quantifying structural coherence in language models. This method extracts structures by projecting data into a latent space of interest and then compares the structures in model generations to human-written text. This quantitative measure of structural coherence enables us to identify structural issues in language models and reveals that structural coherence does not fully correlate with surface fluency. In the second part of the talk, I will present my research on improving structure modeling in language models. I will introduce a global model that scores the overall structure of text, in addition to the traditional language model that scores text by scoring each local word. The traditional language model excels at surface-level modeling, while the introduced global model specializes in structure modeling. I will
demonstrate that the proposed model leads to improvements in both local fluency and structural coherence. To conclude, I will outline my future plans to extend my research into extracting structured representations of text using language models. These representations hold the potential to be applied across a wide array of downstream tasks.
Bio:
Yuntian Deng is a PhD student at Harvard University, advised by Professors Alexander Rush and Stuart Shieber. His research focuses on analyzing and improving structure modeling in language models. He is also a key contributor to several open-source projects, including OpenNMT, Image-to-LaTeX, and LaTeX-to-Image.
Yuntian is the recipient of an Nvidia Fellowship, a Baidu Fellowship, and multiple awards for his research, including the University of Chicago Rising Stars in Data Science, the ACL 2017 Best Demo Paper Runner-Up, the ACM Gordon Bell Special Prize for Covid Research, the Impact Award from Argonne National Lab, and
the DAC 2020 Best Paper.
Abstract: We are approaching a future where text generation technologies will allow us to generate texts that are not only fluent at a surface level, but also coherent in their overall structure. To enable this future, my research
focuses on evaluating and improving structure modeling in language models. In the first part of this talk, I will introduce a method for quantifying structural coherence in language models. This method extracts structures by projecting data into a latent space of interest and then compares the structures in model generations to human-written text. This quantitative measure of structural coherence enables us to identify structural issues in language models and reveals that structural coherence does not fully correlate with surface fluency. In the second part of the talk, I will present my research on improving structure modeling in language models. I will introduce a global model that scores the overall structure of text, in addition to the traditional language model that scores text by scoring each local word. The traditional language model excels at surface-level modeling, while the introduced global model specializes in structure modeling. I will
demonstrate that the proposed model leads to improvements in both local fluency and structural coherence. To conclude, I will outline my future plans to extend my research into extracting structured representations of text using language models. These representations hold the potential to be applied across a wide array of downstream tasks.
Bio:
Yuntian Deng is a PhD student at Harvard University, advised by Professors Alexander Rush and Stuart Shieber. His research focuses on analyzing and improving structure modeling in language models. He is also a key contributor to several open-source projects, including OpenNMT, Image-to-LaTeX, and LaTeX-to-Image.
Yuntian is the recipient of an Nvidia Fellowship, a Baidu Fellowship, and multiple awards for his research, including the University of Chicago Rising Stars in Data Science, the ACL 2017 Best Demo Paper Runner-Up, the ACM Gordon Bell Special Prize for Covid Research, the Impact Award from Argonne National Lab, and
the DAC 2020 Best Paper.
