Uploaded May 2023 | Updated September 2026, 11 hours ago
Abstract: Long-form text generation is an emerging AI technology that has seen extensive recent interest due to the success of ChatGPT. However, several key challenges need to be addressed before long-form generation
systems can be practically deployed at scale. In this talk, I will present my research that focuses on first identifying such challenges, and then designing algorithms to address them. I will start by describing a
retrieval-augmented system built for long-form question answering, which achieves a new state-of-the-art on the ELI5 benchmark. However, a careful empirical analysis reveals two critical issues: (1) models do not
condition their outputs on retrieved evidence articles, and instead hallucinate information from their parametric knowledge; and (2) there is an inherent difficulty in human evaluation of long-form generations. To improve the input/output consistency of generated text (issue #1), I will describe our model RankGen, which is a 1.2B parameter encoder trained with large-scale contrastive learning on documents. RankGen significantly outperforms competing long-form text generation methods in terms of automatic and human evaluation, generating text more faithful to the input. Next, I will describe our efforts to improve human evaluation of long-form generation (issue #2) by proposing the LongEval guidelines. LongEval is a set of
three simple empirically-motivated ideas to make human evaluation of long-form generation more consistent, less expensive, and cognitively easier for evaluators. Finally, I will conclude by discussing a few future directions in long-form generation that I am excited about.
Bio: Kalpesh Krishna is a fifth year PhD candidate at the University of Massachusetts Amherst, where he is advised by Prof. Mohit Iyyer. His primary research interest is in different aspects of long-form text generation. Before coming to UMass, he was an undergraduate student at IIT Bombay, advised by Prof. Preethi Jyothi. He has previously interned at Google, Allen Institute for AI and TTI-Chicago. He is supported by the Google PhD Fellowship, and his research was recently awarded with an outstanding paper award in EACL 2023.
Abstract: Long-form text generation is an emerging AI technology that has seen extensive recent interest due to the success of ChatGPT. However, several key challenges need to be addressed before long-form generation
systems can be practically deployed at scale. In this talk, I will present my research that focuses on first identifying such challenges, and then designing algorithms to address them. I will start by describing a
retrieval-augmented system built for long-form question answering, which achieves a new state-of-the-art on the ELI5 benchmark. However, a careful empirical analysis reveals two critical issues: (1) models do not
condition their outputs on retrieved evidence articles, and instead hallucinate information from their parametric knowledge; and (2) there is an inherent difficulty in human evaluation of long-form generations. To improve the input/output consistency of generated text (issue #1), I will describe our model RankGen, which is a 1.2B parameter encoder trained with large-scale contrastive learning on documents. RankGen significantly outperforms competing long-form text generation methods in terms of automatic and human evaluation, generating text more faithful to the input. Next, I will describe our efforts to improve human evaluation of long-form generation (issue #2) by proposing the LongEval guidelines. LongEval is a set of
three simple empirically-motivated ideas to make human evaluation of long-form generation more consistent, less expensive, and cognitively easier for evaluators. Finally, I will conclude by discussing a few future directions in long-form generation that I am excited about.
Bio: Kalpesh Krishna is a fifth year PhD candidate at the University of Massachusetts Amherst, where he is advised by Prof. Mohit Iyyer. His primary research interest is in different aspects of long-form text generation. Before coming to UMass, he was an undergraduate student at IIT Bombay, advised by Prof. Preethi Jyothi. He has previously interned at Google, Allen Institute for AI and TTI-Chicago. He is supported by the Google PhD Fellowship, and his research was recently awarded with an outstanding paper award in EACL 2023.
