Uploaded July 2025 | Updated September 2026, 3 weeks ago
đź“Ś Description
🎮 What if measuring AGI could be more like playing a game?
Greg Camerad, President of the ARC Prize Foundation, breaks down how their groundbreaking third generation benchmark — ARC AGI 3 — will transform how we measure general intelligence.
🔍 Highlights:
-Why classic static benchmarks fall short at capturing real generalization
-How ARC’s new approach uses interactive reasoning benchmarks with 100+ novel games
-The surprising philosophy of measuring skill acquisition efficiency (Francois Chollet’s elegant idea)
-What made the Atari benchmark phase flawed — and how this is different
-The roadmap for launching this new benchmark (with APIs & contests!)
-Why these tests could finally tell us if an AI is truly learning on the fly
🚀 Whether you're an ML researcher, an AI skeptic, or just fascinated by the path to AGI, this talk is a must-watch.
⏱ Timestamps
00:03 - Intro: Why measuring general intelligence is about to get more fun
0:10 - Claude & Gemini playing Pokemon: Why it’s NOT AGI yet
0:56 - Who is Greg Camerad & ARC Prize: A northstar for open AGI
1:35 - ARC’s unique benchmark philosophy: Humans as the target, because humans = only known proof of general intelligence
2:23 - Defining intelligence: McCarthy & Chollet’s views
3:48 - Skill acquisition efficiency: Learn new things & show it — that’s the real test
4:43 - How ARC AGI 1 & 2 worked: Learning novel transformations, 1000+ tiny skills
5:55 - Why static benchmarks can’t test long-term, open-ended learning
6:54 - Rich Sutton’s “era of experience”: Why agents need to collect their own data
7:52 - ARC’s solution: Interactive reasoning benchmarks — long horizon, multi-turn
8:47 - Why games are the perfect medium for testing intelligence
9:46 - The problems with Atari benchmarks: Dense rewards, unlimited sampling, overfit developers
10:47 - The dream benchmark: 100 games, novel to AI & developers, sparse rewards, no instructions
12:36 - Ensuring zero cultural knowledge: Only core knowledge priors (objects, geometry, agency)
13:50 - Humans still uniquely good at interrogating failures & improving — can AI close this gap?
14:19 - Measuring skill acquisition efficiency in actions, not static scores
14:53 - Timeline: Preview games & API coming July 17, full 100-game launch Q1 next year
15:43 - How you can help: Philanthropy, building agents, or designing novel games
16:52 - Rules for games: Fun for humans, no cultural symbols, no overlapping skills
17:55 - Deep question: But what about human dopamine loops — why do agents even act?
18:23 - Setting rewards & alignment: A question for upstream model builders
18:55 - How to define clear rewards even in chaotic or competitive games
#AGI, #artificialintelligence, #ARCPrize
đź“Ś Description
🎮 What if measuring AGI could be more like playing a game?
Greg Camerad, President of the ARC Prize Foundation, breaks down how their groundbreaking third generation benchmark — ARC AGI 3 — will transform how we measure general intelligence.
🔍 Highlights:
-Why classic static benchmarks fall short at capturing real generalization
-How ARC’s new approach uses interactive reasoning benchmarks with 100+ novel games
-The surprising philosophy of measuring skill acquisition efficiency (Francois Chollet’s elegant idea)
-What made the Atari benchmark phase flawed — and how this is different
-The roadmap for launching this new benchmark (with APIs & contests!)
-Why these tests could finally tell us if an AI is truly learning on the fly
🚀 Whether you're an ML researcher, an AI skeptic, or just fascinated by the path to AGI, this talk is a must-watch.
⏱ Timestamps
00:03 - Intro: Why measuring general intelligence is about to get more fun
0:10 - Claude & Gemini playing Pokemon: Why it’s NOT AGI yet
0:56 - Who is Greg Camerad & ARC Prize: A northstar for open AGI
1:35 - ARC’s unique benchmark philosophy: Humans as the target, because humans = only known proof of general intelligence
2:23 - Defining intelligence: McCarthy & Chollet’s views
3:48 - Skill acquisition efficiency: Learn new things & show it — that’s the real test
4:43 - How ARC AGI 1 & 2 worked: Learning novel transformations, 1000+ tiny skills
5:55 - Why static benchmarks can’t test long-term, open-ended learning
6:54 - Rich Sutton’s “era of experience”: Why agents need to collect their own data
7:52 - ARC’s solution: Interactive reasoning benchmarks — long horizon, multi-turn
8:47 - Why games are the perfect medium for testing intelligence
9:46 - The problems with Atari benchmarks: Dense rewards, unlimited sampling, overfit developers
10:47 - The dream benchmark: 100 games, novel to AI & developers, sparse rewards, no instructions
12:36 - Ensuring zero cultural knowledge: Only core knowledge priors (objects, geometry, agency)
13:50 - Humans still uniquely good at interrogating failures & improving — can AI close this gap?
14:19 - Measuring skill acquisition efficiency in actions, not static scores
14:53 - Timeline: Preview games & API coming July 17, full 100-game launch Q1 next year
15:43 - How you can help: Philanthropy, building agents, or designing novel games
16:52 - Rules for games: Fun for humans, no cultural symbols, no overlapping skills
17:55 - Deep question: But what about human dopamine loops — why do agents even act?
18:23 - Setting rewards & alignment: A question for upstream model builders
18:55 - How to define clear rewards even in chaotic or competitive games
#AGI, #artificialintelligence, #ARCPrize
