Uploaded April 2025 | Updated September 2026, 3 weeks ago
The promise of AI agents has led to claims of imminent and rapid adoption across fields. Companies have even promised to build AI agents that can automate all legal and scientific tasks. At the same time, there are concerns about their misuse leading to catastrophic risks such as bio and cybersecurity risks.
In this talk, Sayash Kapoor goes over three case studies to foreground the importance of evidence-based AI analysis. First, while AI has been claimed to automate all of science, existing adoption has been plagued by severe reproducibility failures that lead to overoptimistic results across dozens of fields. Recent empirical work shows that current models fall well short of accomplishing far simpler tasks, such as reproducing a paper’s results even when the code and data are provided. Second, for legal applications, tasks that would lead to the most significant changes to the legal profession are also the ones most prone to overoptimism about AI capabilities, as they are harder to evaluate. Third, for analyzing safety risks, it is important to analyze the marginal risk of AI over and above existing technology to evaluate the effectiveness of policy interventions. Kapoor concludes with a discussion of how to effectively conduct empirical analysis of AI.
The promise of AI agents has led to claims of imminent and rapid adoption across fields. Companies have even promised to build AI agents that can automate all legal and scientific tasks. At the same time, there are concerns about their misuse leading to catastrophic risks such as bio and cybersecurity risks.
In this talk, Sayash Kapoor goes over three case studies to foreground the importance of evidence-based AI analysis. First, while AI has been claimed to automate all of science, existing adoption has been plagued by severe reproducibility failures that lead to overoptimistic results across dozens of fields. Recent empirical work shows that current models fall well short of accomplishing far simpler tasks, such as reproducing a paper’s results even when the code and data are provided. Second, for legal applications, tasks that would lead to the most significant changes to the legal profession are also the ones most prone to overoptimism about AI capabilities, as they are harder to evaluate. Third, for analyzing safety risks, it is important to analyze the marginal risk of AI over and above existing technology to evaluate the effectiveness of policy interventions. Kapoor concludes with a discussion of how to effectively conduct empirical analysis of AI.
