Machine Learning Street Talk | Chollet's ARC Challenge + Current Winners @MachineLearningStreetTalk | Uploaded June 2024 | Updated October 2024, 2 hours ago.
The ARC Challenge, created by Francois Chollet, tests how well AI systems can generalize from a few examples in a grid-based intelligence test. We interview the current winners of the ARC Challenge—Jack Cole, Mohammed Osman and their collaborator Michael Hodel. They discuss how they tackled ARC (Abstraction and Reasoning Corpus) using language models. We also discuss the new "50%" public set approach announced today from Redwood Research (Ryan Greenblatt).
Jack and Mohammed explain their winning approach, which involves fine-tuning a language model on a large, specifically-generated dataset and then doing additional fine-tuning at test-time, a technique known in this context as "active inference". They use various strategies to represent the data for the language model and believe that with further improvements, the accuracy could reach above 50%. Michael talks about his work on generating new ARC-like tasks to help train the models.
They also debate whether their methods stay true to the "spirit" of Chollet's measure of intelligence. Despite some concerns, they agree that their solutions are promising and adaptable for other similar problems.
Note:
Jack's team is still the current official winner at 33% on the private set. Ryan's entry is not on the private leaderboard or eligible.
Chollet invented ARC in 2019 (not 2017 as stated)
"Ryan's entry is not a new state of the art. We don't know exactly how well it does since it was only evaluated on 100 tasks from the evaluation set and does 50% on those, reportedly. Meanwhile Jacks team i.e. MindsAI's solution does 54% on the entire eval set and it is seemingly possible to do 60-70% with an ensemble"
Jack Cole:
https://x.com/Jcole75Cole
https://lab42.global/community-interview-jack-cole/
Mohamed Osman:
Mohamed is looking to do a PhD in AI/ML, can you help him?
Email: mothman198@outlook.com
linkedin.com/in/mohamedosman1905
Michael Hodel:
arxiv.org/pdf/2404.07353v1
linkedin.com/in/michael-hodel
https://x.com/bayesilicon
github.com/michaelhodel
Getting 50% (SoTA) on ARC-AGI with GPT-4o - Ryan Greenblatt
redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt
Neural networks for abstraction and reasoning: Towards broad generalization in machines [Mikel Bober-Irizar, Soumya Banerjee]
arxiv.org/pdf/2402.03507
Measure of intelligence:
arxiv.org/abs/1911.01547
I think the audio levelling might be a bit off on this for the intro especially, I fixed it on the audio podcast version - sorry if it's annoying.
Pod version: podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/New-50-ARC-result-and-current-winners-interviewed-e2l1prl
TOC (autogenerated):
00:00:00 Introduction
00:03:00 Francois Chollet's Intelligence Concept
00:08:00 Human Collaboration
00:15:00 ARC Tasks and Symbolic AI
00:27:00 Evaluation Techniques
00:35:23 (Main Interview) Competitors and Approaches
00:40:00 Meta Learning Challenges
00:48:00 System 1 vs System 2
01:00:00 Inductive Priors and Symbols
01:18:00 Methodologies Comparison
01:25:00 Training Data Size Impact
01:35:00 Generalization Issues
01:47:00 Techniques for AI Applications
01:56:00 Model Efficiency and Scalability
02:10:00 Task Specificity and Generalization
02:13:00 Summary
The ARC Challenge, created by Francois Chollet, tests how well AI systems can generalize from a few examples in a grid-based intelligence test. We interview the current winners of the ARC Challenge—Jack Cole, Mohammed Osman and their collaborator Michael Hodel. They discuss how they tackled ARC (Abstraction and Reasoning Corpus) using language models. We also discuss the new "50%" public set approach announced today from Redwood Research (Ryan Greenblatt).
Jack and Mohammed explain their winning approach, which involves fine-tuning a language model on a large, specifically-generated dataset and then doing additional fine-tuning at test-time, a technique known in this context as "active inference". They use various strategies to represent the data for the language model and believe that with further improvements, the accuracy could reach above 50%. Michael talks about his work on generating new ARC-like tasks to help train the models.
They also debate whether their methods stay true to the "spirit" of Chollet's measure of intelligence. Despite some concerns, they agree that their solutions are promising and adaptable for other similar problems.
Note:
Jack's team is still the current official winner at 33% on the private set. Ryan's entry is not on the private leaderboard or eligible.
Chollet invented ARC in 2019 (not 2017 as stated)
"Ryan's entry is not a new state of the art. We don't know exactly how well it does since it was only evaluated on 100 tasks from the evaluation set and does 50% on those, reportedly. Meanwhile Jacks team i.e. MindsAI's solution does 54% on the entire eval set and it is seemingly possible to do 60-70% with an ensemble"
Jack Cole:
https://x.com/Jcole75Cole
https://lab42.global/community-interview-jack-cole/
Mohamed Osman:
Mohamed is looking to do a PhD in AI/ML, can you help him?
Email: mothman198@outlook.com
linkedin.com/in/mohamedosman1905
Michael Hodel:
arxiv.org/pdf/2404.07353v1
linkedin.com/in/michael-hodel
https://x.com/bayesilicon
github.com/michaelhodel
Getting 50% (SoTA) on ARC-AGI with GPT-4o - Ryan Greenblatt
redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt
Neural networks for abstraction and reasoning: Towards broad generalization in machines [Mikel Bober-Irizar, Soumya Banerjee]
arxiv.org/pdf/2402.03507
Measure of intelligence:
arxiv.org/abs/1911.01547
I think the audio levelling might be a bit off on this for the intro especially, I fixed it on the audio podcast version - sorry if it's annoying.
Pod version: podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/New-50-ARC-result-and-current-winners-interviewed-e2l1prl
TOC (autogenerated):
00:00:00 Introduction
00:03:00 Francois Chollet's Intelligence Concept
00:08:00 Human Collaboration
00:15:00 ARC Tasks and Symbolic AI
00:27:00 Evaluation Techniques
00:35:23 (Main Interview) Competitors and Approaches
00:40:00 Meta Learning Challenges
00:48:00 System 1 vs System 2
01:00:00 Inductive Priors and Symbols
01:18:00 Methodologies Comparison
01:25:00 Training Data Size Impact
01:35:00 Generalization Issues
01:47:00 Techniques for AI Applications
01:56:00 Model Efficiency and Scalability
02:10:00 Task Specificity and Generalization
02:13:00 Summary
