Uploaded November 2024 | Updated September 2026, 2 weeks ago
How close are #ai systems to conducting autonomous AI research? In this talk, Lawrence Chan from METR shares insights from evaluations of Anthropic's #Claude3.5 Sonnet and #OpenAI’s o1 models. Discover how these AI models compare to human #mlengineers and explore the potential for scaling laws to measure AI's independent R&D capabilities.
Timestamps:
0:00 Introduction
0:11 What is METR and what does it do?
6:12 A high level story of AI risk: people build AI with dangerous capabilities and AI causes unacceptably catastrophic outcomes
8:10 Desiderata for evaluations
11:40 Why AI R&D?
20:37 AI R&D Results
22:30 Results (WIP)
25:40 Qualitative analysis/ transcript review
28:50 Problems: saturation, correspondence, small n, qualitative results
33:00 Main takeaways
#airesearch #autonomousai #machinelearning #artificialintelligence #claude3 #openai #aicapabilities #MLengineers #futureofai #ScalingLaws #aiscientist #airesearch #scientificdiscovery #futureofAI #AIandscience #researchautomation #artificialgeneralintelligence #transformers #deeplearning #technology #science #techtalk #techtalks #largelanguagemodels #llms #anthropic #openai #gpt4 #gpt #claude #claude3
Social Links:
Newsletter: buzzrobot.substack.com
X: https://x.com/sopharicks
Slack: join.slack.com/t/buzzrobot/shared_invite/zt-2s067rv7n-guPIMGe62rbp9ncxdnOUfQ
How close are #ai systems to conducting autonomous AI research? In this talk, Lawrence Chan from METR shares insights from evaluations of Anthropic's #Claude3.5 Sonnet and #OpenAI’s o1 models. Discover how these AI models compare to human #mlengineers and explore the potential for scaling laws to measure AI's independent R&D capabilities.
Timestamps:
0:00 Introduction
0:11 What is METR and what does it do?
6:12 A high level story of AI risk: people build AI with dangerous capabilities and AI causes unacceptably catastrophic outcomes
8:10 Desiderata for evaluations
11:40 Why AI R&D?
20:37 AI R&D Results
22:30 Results (WIP)
25:40 Qualitative analysis/ transcript review
28:50 Problems: saturation, correspondence, small n, qualitative results
33:00 Main takeaways
#airesearch #autonomousai #machinelearning #artificialintelligence #claude3 #openai #aicapabilities #MLengineers #futureofai #ScalingLaws #aiscientist #airesearch #scientificdiscovery #futureofAI #AIandscience #researchautomation #artificialgeneralintelligence #transformers #deeplearning #technology #science #techtalk #techtalks #largelanguagemodels #llms #anthropic #openai #gpt4 #gpt #claude #claude3
Social Links:
Newsletter: buzzrobot.substack.com
X: https://x.com/sopharicks
Slack: join.slack.com/t/buzzrobot/shared_invite/zt-2s067rv7n-guPIMGe62rbp9ncxdnOUfQ
