Uploaded October 2026 | Updated November 2026, 3 days ago
Mohit Iyyer (University of Maryland, College Park)
https://simons.berkeley.edu/talks/mohit-iyyer-university-maryland-college-park-2026-10-09
Trustworthy AI: From Hallucinations to Reliable Autonomy
In this talk, I consider the growing problem of evaluating the factuality of long-form texts generated by LLMs (e.g., articles, essays, books): as the length and complexity of long-form writing tasks increases, how do we even measure progress? First, I propose a high-level framework (applicable to both human and automatic evaluation) that decomposes a long-form text into simpler atomic units before then evaluating each unit on a specific aspect. I demonstrate the framework's effectiveness at evaluating factuality and faithfulness on tasks such as long-form question answering and book summarization, and also discuss how we can speed up this pipeline to scale up evaluations. Next, I'll discuss evaluation in the context of agentic writing pipelines, in which models can use tools and write/execute code to aid the production of long-form reports, focusing on our recent AnalystBench project. I'll conclude with some open problems: (1) reliance on faulty retrievers / search engines; (2) verifying complex, highly-compositional claims; and (3) evaluating interpretive claims rather than fully objective ones.
Mohit Iyyer (University of Maryland, College Park)
https://simons.berkeley.edu/talks/mohit-iyyer-university-maryland-college-park-2026-10-09
Trustworthy AI: From Hallucinations to Reliable Autonomy
In this talk, I consider the growing problem of evaluating the factuality of long-form texts generated by LLMs (e.g., articles, essays, books): as the length and complexity of long-form writing tasks increases, how do we even measure progress? First, I propose a high-level framework (applicable to both human and automatic evaluation) that decomposes a long-form text into simpler atomic units before then evaluating each unit on a specific aspect. I demonstrate the framework's effectiveness at evaluating factuality and faithfulness on tasks such as long-form question answering and book summarization, and also discuss how we can speed up this pipeline to scale up evaluations. Next, I'll discuss evaluation in the context of agentic writing pipelines, in which models can use tools and write/execute code to aid the production of long-form reports, focusing on our recent AnalystBench project. I'll conclude with some open problems: (1) reliance on faulty retrievers / search engines; (2) verifying complex, highly-compositional claims; and (3) evaluating interpretive claims rather than fully objective ones.










