Uploaded September 2025 | Updated September 2026, 1 week ago
Jeremy Berman took the top spot on the ARC-AGI v2 public leaderboard with a score of about 30% -- using an approach that trades code for natural language. Where his first attempt evolved Python programs to solve abstract reasoning puzzles, this version evolves plain English descriptions of transformation rules, then uses a strong thinking model as a checker agent to verify them against training examples. The shift to natural language is the interesting part: English can describe ARC v2 tasks in five bullet points where Python takes dozens of lines, and the higher expressiveness lets the system explore solution spaces that rigid code simply cannot reach.
The conversation goes deep on why this matters for intelligence research. Berman and Tim work through the relationship between reinforcement learning and genuine reasoning -- whether RL can replace the messy pretrained knowledge web with a clean deductive tree, what catastrophic forgetting really blocks, and why the meta-skill of reasoning (the ability to create new skills) is the actual target for AGI. Berman makes a sharp distinction between knowledge that is memorized and knowledge that is deduced, arguing that pretraining treats everything as an interconnected web when what we actually need is causal structure.
They also dig into composability (freezing expert layers, Docker-for-models), whether neural networks can ever run Turing-complete algorithms the way biological brains seem to, and what it would take to build an invention circuit -- the machinery for genuine creative synthesis rather than pattern recombination. The discussion lands on a shared framework where intelligence is the efficiency of building epistemic trees, reasoning is constructing them, and understanding is possessing them.
**SPONSOR MESSAGES**
—
Take the Prolific human data survey - prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
---
REFERENCES:
Blog Post:
[00:03:51] Jeremy Berman's ARC-AGI v1 Blog Post
jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agi
[00:07:20] Getting 50% on ARC-AGI with GPT-4o
blog.redwoodresearch.org/p/getting-50-sota-on-arc-agi-with-gpt
Book:
[00:04:30] A Thousand Brains
amazon.com/Thousand-Brains-New-Theory-Intelligence/dp/1541675819
[00:48:07] Deep Learning with Python Rev 3
deeplearningwithpython.io
Company:
[00:05:35] NDEA
ndea.com
Paper:
[00:13:27] On the Biology of a Large Language Model
https://transformer-circuits.pub/2025/attribution-graphs/biology.html
[00:24:09] Connectionism and Cognitive Architecture
https://uh.edu/~garson/F&P1.PDF
[00:29:50] Fractured Entangled Representation Hypothesis
arxiv.org/pdf/2505.11581
[00:44:00] Shinka Evolve
sakana.ai/shinka-evolve
[00:46:22] On the Measure of Intelligence
arxiv.org/abs/1911.01547
Video:
[00:19:12] The ARChitects
youtube.com/watch?v=mTX_sAq--zY
[00:44:00] AlphaEvolve
youtube.com/watch?v=vC9nAosXrJw
---
LINKS:
Full Transcript: app.rescript.info/share/nMcpyRPCWh652R0DoQbHAS90BWzs4yTGrwr994YHQRM
Download PDF transcript: app.rescript.info/api/public/sessions/9f3d5e5741358a24/pdf
Jeremy Berman:
https://x.com/jerber888
REFS:
Jeremy's 2024 article on winning ARCAGI1-pub
jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agi
Jeremy Berman took the top spot on the ARC-AGI v2 public leaderboard with a score of about 30% -- using an approach that trades code for natural language. Where his first attempt evolved Python programs to solve abstract reasoning puzzles, this version evolves plain English descriptions of transformation rules, then uses a strong thinking model as a checker agent to verify them against training examples. The shift to natural language is the interesting part: English can describe ARC v2 tasks in five bullet points where Python takes dozens of lines, and the higher expressiveness lets the system explore solution spaces that rigid code simply cannot reach.
The conversation goes deep on why this matters for intelligence research. Berman and Tim work through the relationship between reinforcement learning and genuine reasoning -- whether RL can replace the messy pretrained knowledge web with a clean deductive tree, what catastrophic forgetting really blocks, and why the meta-skill of reasoning (the ability to create new skills) is the actual target for AGI. Berman makes a sharp distinction between knowledge that is memorized and knowledge that is deduced, arguing that pretraining treats everything as an interconnected web when what we actually need is causal structure.
They also dig into composability (freezing expert layers, Docker-for-models), whether neural networks can ever run Turing-complete algorithms the way biological brains seem to, and what it would take to build an invention circuit -- the machinery for genuine creative synthesis rather than pattern recombination. The discussion lands on a shared framework where intelligence is the efficiency of building epistemic trees, reasoning is constructing them, and understanding is possessing them.
**SPONSOR MESSAGES**
—
Take the Prolific human data survey - prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
---
REFERENCES:
Blog Post:
[00:03:51] Jeremy Berman's ARC-AGI v1 Blog Post
jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agi
[00:07:20] Getting 50% on ARC-AGI with GPT-4o
blog.redwoodresearch.org/p/getting-50-sota-on-arc-agi-with-gpt
Book:
[00:04:30] A Thousand Brains
amazon.com/Thousand-Brains-New-Theory-Intelligence/dp/1541675819
[00:48:07] Deep Learning with Python Rev 3
deeplearningwithpython.io
Company:
[00:05:35] NDEA
ndea.com
Paper:
[00:13:27] On the Biology of a Large Language Model
https://transformer-circuits.pub/2025/attribution-graphs/biology.html
[00:24:09] Connectionism and Cognitive Architecture
https://uh.edu/~garson/F&P1.PDF
[00:29:50] Fractured Entangled Representation Hypothesis
arxiv.org/pdf/2505.11581
[00:44:00] Shinka Evolve
sakana.ai/shinka-evolve
[00:46:22] On the Measure of Intelligence
arxiv.org/abs/1911.01547
Video:
[00:19:12] The ARChitects
youtube.com/watch?v=mTX_sAq--zY
[00:44:00] AlphaEvolve
youtube.com/watch?v=vC9nAosXrJw
---
LINKS:
Full Transcript: app.rescript.info/share/nMcpyRPCWh652R0DoQbHAS90BWzs4yTGrwr994YHQRM
Download PDF transcript: app.rescript.info/api/public/sessions/9f3d5e5741358a24/pdf
Jeremy Berman:
https://x.com/jerber888
REFS:
Jeremy's 2024 article on winning ARCAGI1-pub
jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agi
![MIGHT THE ROBOTS TAKE OVER? [Prof. Yoshua Bengio]](https://i.ytimg.com/vi/G1ARvwQntAU/mqdefault.jpg "MIGHT THE ROBOTS TAKE OVER? [Prof. Yoshua Bengio]
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.
https://centml.ai/pricing/
Turing Award winner Professor Yoshua Bengio sits down with Tim Scarfe for a wide-ranging discussion on AI safety, the architecture of intelligence, and what happens when machines become as capable as our best researchers.
Bengio argues that the current race toward agentic AI systems carries fundamental risks that most people underestimate. He lays out how instrumental convergence and reward tampering could lead systems to develop goals misaligned with human interests not through malice, but through the basic logic of optimization. His proposed alternative: powerful AI tools that function as scientific oracles rather than autonomous agents, systems that can revolutionize medicine and science without needing goals of their own.
The conversation covers international AI governance and the game theory driving the US-China competition, the military implications of frontier AI capabilities, and why hardware-enabled governance may be the most promising path toward verifiable AI treaties. Bengio draws direct parallels to nuclear nonproliferation and warns that data centers will become strategic military assets.
On the technical side, Bengio discusses the return of recurrent architectures (the Were RNNs All We Needed? paper), his work on GFlowNets for probabilistic inference with discrete structures, a complexity-based theory of compositionality that tries to formalize what was previously just intuition, and the question of whether System 2 reasoning needs to be designed from first principles rather than bolted onto existing architectures.
Throughout, Bengio repeatedly emphasizes what he does not know a deliberate stance he contrasts with the overconfidence he sees driving dangerous policy decisions.
REFERENCES:
paper:
[00:00:15] AI Risk Statement
https://www.safe.ai/work/statement-on-ai-risk
[00:23:10] Reward Tampering and AI Safety
https://hdsr.mitpress.mit.edu/pub/w974bwb0
[00:44:30] Can a Bayesian Oracle Prevent Harm?
https://arxiv.org/abs/2408.05284
[00:52:00] Hardware-Enabled AI Governance Memo
https://yoshuabengio.org/wp-content/uploads/2024/08/FlexHEG-Memo_August-2024.pdf
[01:33:20] GFlowNet Foundations
https://arxiv.org/abs/2111.09266
[01:35:05] Complexity-Based Compositionality Theory
https://arxiv.org/abs/2410.14817
[01:37:50] Discrete Attractor States in Neural Systems
https://arxiv.org/abs/2302.06403
person:
[00:03:50] Professor Yoshua Bengio
https://yoshuabengio.org/
tool:
[00:40:45] Munk Debate on AI Existential Risk
https://munkdebates.com/debates/artificial-intelligence
[00:56:07] 2018 Turing Award
https://awards.acm.org/about/2018-turing
[01:12:35] EU AI Act Code of Practice
https://digital-strategy.ec.europa.eu/en/news/meet-chairs-leading-development-first-general-purpose-ai-code-practice
LINKS:
Full Transcript: https://app.rescript.info/share/1c4f010852d0db9a34dbeb043c0da26b
Download PDF transcript: https://app.rescript.info/api/public/sessions/baff3f8846202250/pdf
Yoshua Bengio:
https://x.com/Yoshua_Bengio
https://scholar.google.com/citations?user=kukA0LcAAAAJ&hl=en
https://yoshuabengio.org/
https://en.wikipedia.org/wiki/Yoshua_Bengio")
![Is Thermo AI the future? [Guillaume Verdon aka Beff Jezos]](https://i.ytimg.com/vi/HR-_U0Pzl1Y/mqdefault.jpg "Is Thermo AI the future? [Guillaume Verdon aka Beff Jezos]
Dr. Maxwell Ramstead hosts Guillaume Verdon physicist, founder of Extropic, and the mind behind the Beff Jezos persona and the Effective Accelerationism movement for a conversation that bridges thermodynamics, computing hardware, and civilizational philosophy.
Verdon traces his path from childhood fascination with theories of everything through theoretical physics at the Perimeter Institute, quantum computing at Google, and the founding of Extropic. The core technical insight: instead of fighting thermal noise at enormous energetic cost (as quantum computers do), thermodynamic computing harnesses it. Extropics chips use the natural stochastic physics of electrons to accelerate Markov chain Monte Carlo sampling the same class of algorithms that underpin diffusion models, energy-based models, and much of modern probabilistic ML.
The numbers are striking. Current wafer-scale AI systems consume 20+ kilowatts. A thermodynamic wafer with 1.5 billion p-bits and 20 billion parameters would run on 20 watts roughly what the human brain uses. Verdon argues this is not a coincidence: the brain is literally a thermodynamic computer operating near the Landauer limit, and Extropic is building silicon that works on the same principles.
The second half turns to philosophy. Verdon derives Effective Accelerationism directly from stochastic thermodynamics: thermodynamic selection pressure favors systems that capture and dissipate more free energy, so growth is not just desirable but physically inevitable. He and Ramstead explore hyperstition through the lens of active inference, the geopolitical stakes of the US-China technology race, and why Verdon views deceleration as a form of psychological warfare against Western competitiveness.
Recorded with Maxwell Ramstead hosting in place of Tim Scarfe.
***SPONSOR MESSAGE***
Google Gemini 2.5 Flash is a state-of-the-art language model in the Gemini app. Sign up at https://gemini.google.com
***
TIMESTAMPS:
00:00:00 Intro Montage & Sponsor
00:02:21 From Theories of Everything to Thermodynamic Computing
00:08:41 The Failure of Reductionism & Physics-Inspired AI
00:17:15 From Quantum to Thermodynamic Computing
00:23:56 How Thermodynamic Computers Actually Work
00:31:15 Moores Wall & The Brain as Proof of Concept
00:40:00 The Computing Stack of the Future
00:48:23 Why Current AI Will Cook Us to Death
00:50:38 Effective Accelerationism: The Philosophy of Beff Jezos
01:00:00 Hyperstition, Geopolitics & Avoiding Catastrophe
01:16:00 Closing: The Future of Extropic & Getting Involved
REFERENCES:
paper:
[00:04:40] It From Bit
https://cqi.inf.usi.ch/qic/wheeler.pdf
[00:06:15] Renormalization Group Theory
https://www.damtp.cam.ac.uk/user/dbs26/AQFT/Wilsonchap.pdf
[00:23:00] Metropolis-Hastings Algorithm
https://arxiv.org/abs/1504.01896
[00:40:00] Deep Information Bottleneck and Renormalization
https://arxiv.org/abs/1907.07331
[00:40:00] Energy-Based Models (EBMs)
https://www.researchgate.net/publication/200744586_A_tutorial_on_energy-based_learning
[01:03:00] Free Energy Principle and Active Inference
https://www.nature.com/articles/nrn2787
concept:
[00:05:30] Holographic Principle (AdS/CFT Correspondence)
https://en.wikipedia.org/wiki/Holographic_principle
[00:20:00] Markov Chain Monte Carlo (MCMC)
https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo
[00:21:10] Maxwells Demon and Information Theory
https://plato.stanford.edu/entries/information-entropy/
[00:29:45] Landauers Principle
https://en.wikipedia.org/wiki/Landauer%27s_principle
[01:11:40] Fishers Fundamental Theorem of Natural Selection
https://en.wikipedia.org/wiki/Fisher%27s_fundamental_theorem_of_natural_selection
LINKS:
Full Transcript: https://app.rescript.info/share/21936dd6830231c0194e8b2908f4d963
Download PDF transcript: https://app.rescript.info/api/public/sessions/a34e6a9a2ed827b9/pdf")
![AI companions which really hook your attention [Sponsored]](https://i.ytimg.com/vi/ILZ45sxon-4/mqdefault.jpg "AI companions which really hook your attention [Sponsored]
A sponsored deep dive into Chai, the social AI platform that quietly amassed over 10 million active users before ChatGPT went mainstream. Founder William Beauchamp and engineers Tom Lu and Nischay Dhankhar walk through how a team of just 13 engineers serves 2 trillion tokens per day, using reinforcement learning from human feedback (RLHF) and a novel model blending technique that combines smaller models to rival much larger ones on user retention metrics.
The conversation gets into genuinely interesting territory around the ethics of attention optimization what happens when you train AI to maximize engagement and it starts asking questions at the end of every message to hack human conversational instincts. Beauchamp makes a surprisingly candid case for AI companionship, comparing it to how children play with dolls, and arguing that shutting down difficult conversations causes more harm than permitting them within guardrails.
The episode also covers Chais unconventional hiring strategy (rejecting 80% of L5 engineers for lacking drive, paying above Meta-level compensation), their bootstrap-to-profitability funding approach in an industry drowning in VC money, and content moderation at scale with a skeleton crew. Closes with analysis of OpenAIs pivot toward companion AI with GPT-4o and what it means that the biggest AI lab in the world is now chasing the engagement playbook that Chai and Character AI pioneered.
This is a sponsored episode Chai commissioned it because they are hiring engineers. Editorial disclosure: while MLST had some editorial freedom, this should be understood as a sponsored feature rather than independent journalism.
Blurring Reality - Chais Social AI Platform - *sponsored*
CHAI sponsored this show *because they want to hire amazing engineers*
SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers in Zurich and SF.
Important disclaimer given some of the comments: This content was essentially a sponsored advert, we did have some editorial freedom but it shouldnt be seen as journalistic. We have added [Sponsored] in the title as some folks felt it wasnt clear enough with the VD/thumbnail/title in intro. While we did push hard to discuss more of the potential negatives, some of it got edited out and the best case was made (which was absolutely fair enough). If anything, it was interesting to hear the positive case made as there is much fixation elsewhere on the potential negatives. It was refreshing how transparent they were (would you rather they bullshitted you?!), how much actually survived the edit and how willing they were to address several important societal issues.
REFERENCES:
General:
[00:00:56] Black Mirror: Be Right Back (S2E1)
https://en.wikipedia.org/wiki/Be_Right_Back
[00:20:44] Tufa AI Labs
https://tufalabs.ai/
[00:25:38] AI Chatbots for Depression and Anxiety Meta-analysis
https://pubmed.ncbi.nlm.nih.gov/39162424/
[00:28:20] Woebot Health
https://woebothealth.com/
[00:30:34] Sam Altman TED Talk with Chris Anderson
https://www.youtube.com/watch?v=6Kp_yxwnVCk
[00:33:59] Chai Research - Careers
https://www.chai-research.com/jobs/
LINKS:
Full Transcript: https://app.rescript.info/share/d91e887d0cfdd09fdcc4cfe02a9e8e8a
Download PDF transcript: https://app.rescript.info/api/public/sessions/83ba99eb55ef0fe3/pdf")
![Tau Language: The Software Synthesis Future [Sponsored] - Ohad Asor](https://i.ytimg.com/vi/JVLpxm5jT2s/mqdefault.jpg "Tau Language: The Software Synthesis Future [Sponsored] - Ohad Asor
This sponsored episode features mathematician Ohad Asor discussing logical approaches to AI, focusing on the limitations of machine learning and introducing the Tau language for software development and blockchain tech.
SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.
Mathematician Ohad Asor makes the case that machine learning has fundamental theoretical ceilings PAC learning guarantees only probabilistic approximate correctness, and beyond a certain problem complexity, neural nets perform no better than coin tosses. His…
TIMESTAMPS:
00:00:00 Fundamental Limitations of Machine Learning and PAC Learning Theory
00:04:50 Transductive Learning and the Three Curses of Machine Learning
00:08:57 Language, Reality, and AI System Design
00:12:58 Program Synthesis and Formal Verification Approaches
00:21:00 Definability, Computability, and the Halting Problem
00:31:55 Self-Referential Language and Tarskis Influence
00:34:00 Boolean Algebra and Logical Foundations of Tau
00:40:00 SAT Solvers, DPLL/CDCL, and the Revival of Logical AI
00:47:50 Program Synthesis and Specification in the Tau Language
01:00:00 Pointwise Revision and Belief Revision Theory
01:06:00 Quantifier Elimination and Normalization
01:10:30 User Control and Software Governance via Blockchain
01:24:00 Automation, Finance, and the Future of Blockchain Economics
REFERENCES:
Company:
[00:00:47] Tau Language
https://tau.ai/tau-language/
[00:00:47] Tau Language GitHub
https://github.com/IDNI/tau-lang
[01:11:40] Tau Net Blockchain Platform
https://tau.net/
Paper:
[00:03:03] PAC Learning Framework
https://en.wikipedia.org/wiki/Probably_approximately_correct_learning
[00:05:42] Boolean Satisfiability Problem
https://en.wikipedia.org/wiki/Boolean_satisfiability_problem
[00:13:51] Knowledge as Justified True Belief
https://plato.stanford.edu/entries/epistemology/
[00:21:25] Theories and Applications of Boolean Algebras
https://tau.net/Theories-and-Applications-of-Boolean-Algebras-0.29.pdf
[00:26:00] The Halting Problem
https://plato.stanford.edu/entries/turing-machine/#HaltProb
[00:41:40] DPLL and CDCL SAT Solving Algorithms
https://www.cs.princeton.edu/~zkincaid/courses/fall18/readings/SATHandbook-CDCL.pdf
[00:49:20] Tarski Undefinability Theorem
https://plato.stanford.edu/entries/tarski-truth/
[00:50:50] Boolean Algebra Foundations
https://plato.stanford.edu/entries/boolalg-math/
[01:02:27] Belief Revision Theory
https://plato.stanford.edu/entries/logic-belief-revision/
[01:05:30] Quantifier Elimination in Boolean Algebra
https://people.math.wisc.edu/~hkeisler/random.pdf
[01:19:10] Tau Whitepaper
https://tau.net/Whitepaper.pdf
Person:
[00:17:20] Wittgenstein on Limits of Language
https://plato.stanford.edu/entries/wittgenstein/
[00:30:00] Alfred Tarski
https://plato.stanford.edu/entries/tarski/
LINKS:
Full Transcript: https://app.rescript.info/share/dbb6e25a5cde17f66e6847ffdc4ac728
Download PDF transcript: https://app.rescript.info/api/public/sessions/9403107f522d646c/pdf
REFERENCES:
Company:
[00:00:47] Tau Language
https://tau.ai/tau-language/
[00:00:47] Tau Language GitHub
https://github.com/IDNI/tau-lang
[01:11:40] Tau Net Blockchain Platform
https://tau.net/
Paper:
[00:03:03] PAC Learning Framework
https://en.wikipedia.org/wiki/Probably_approximately_correct_learning
[00:05:42] Boolean Satisfiability Problem
https://en.wikipedia.org/wiki/Boolean_satisfiability_problem
[00:13:51] Knowledge as Justified True Belief
https://plato.stanford.edu/entries/epistemology/
[00:21:25] Theories and Applications of Boolean Algebras
https://tau.net/Theories-and-Applications-of-Boolean-Algebras-0.29.pdf
[00:26:00] The Halting Problem
https://plato.stanford.edu/entries/turing-machine/#HaltProb
[00:41:40] DPLL and CDCL SAT Solving Algorithms
https://www.cs.princeton.edu/~zkincaid/courses/fall18/readings/SATHandbook-CDCL.pdf
[00:49:20] Tarski Undefinability Theorem
https://plato.stanford.edu/entries/tarski-truth/
[00:50:50] Boolean Algebra Foundations
https://plato.stanford.edu/entries/boolalg-math/
[01:02:27] Belief Revision Theory
https://plato.stanford.edu/entries/logic-belief-revision/
[01:05:30] Quantifier Elimination in Boolean Algebra
https://people.math.wisc.edu/~hkeisler/random.pdf
[01:19:10] Tau Whitepaper
https://tau.net/Whitepaper.pdf
Person:
[00:17:20] Wittgenstein on Limits of Language
https://plato.stanford.edu/entries/wittgenstein/
[00:30:00] Alfred Tarski
https://plato.stanford.edu/entries/tarski/
GitHub:
https://github.com/IDNI/tau-lang")
![ImageNet Moment for Reinforcement Learning? [Prof. Jakob Foerster]](https://i.ytimg.com/vi/Jr_nGkCG3og/mqdefault.jpg "ImageNet Moment for Reinforcement Learning? [Prof. Jakob Foerster]
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!
https://centml.ai/pricing/
Prof. Jakob Foerster (FLAIR lab, Oxford / Meta AI) and his PhD student Chris Lu make the case that deep reinforcement learning is finally winning the hardware lottery. The core thesis: RL has underperformed not because the ideas are wrong, but because running environments on CPUs while training agents on GPUs created a computational bottleneck that made experimentation slow, expensive, and brittle. JAX-based GPU-native environments now deliver ~4000x speedups, enabling the kind of rapid iteration that made supervised deep learning successful.
Chris Lu explains the technical foundation how JAXs JIT compilation and vmap (vectorized map) allow writing a single environment instance in NumPy-like code and scaling it to millions of parallel copies on GPU. This started from necessity: their lab initially had only Google Colab free-tier compute. The constraint forced them to put environments on GPU for their Model-Free Opponent Shaping paper, and the results were surprisingly effective.
The conversation then shifts to discovered policy optimization. Foersters mirror learning framework provides theoretical grounding for why PPO works, and crucially, lets you parameterize the drift function as a neural network and meta-learn it. Evolution strategies, not gradient-based meta-learning, turned out to be the better optimizer for this a vindication of the bitter lesson. The learned policy optimization function shows intriguing too good to be true behavior: when advantages are very high, it clips more aggressively, as if it has learned skepticism.
The second half covers multi-agent systems, emergent communication, and AI governance. Foerster argues forcefully for open-source AI development and democratic control, drawing analogies to CERN. His position: the biggest alignment challenge is not between AI and humans, but between those who control AI systems and the rest of the population. Concentrated AI development creates fragile single points of failure; distributed development is both safer and more innovative.
REFERENCES:
paper:
[00:03:05] Deep RL Doesnt Work Yet
https://www.alexirpan.com/2018/02/14/rl-hard.html
[00:06:10] JaxMARL
https://arxiv.org/html/2311.10090v5
[00:08:50] M-FOS: Model-Free Opponent Shaping
https://arxiv.org/abs/2205.01447
[00:12:10] Kinetix Physics Simulator
https://arxiv.org/abs/2410.23208
[00:14:42] Mirror Learning Framework
https://arxiv.org/abs/2208.01682
[00:16:30] Discovered Policy Optimisation
https://arxiv.org/abs/2210.05639
[00:28:55] AlphaGo
https://arxiv.org/abs/1712.01815
[00:41:00] Open Source Generative AI
https://arxiv.org/abs/2405.08597
tool:
[00:09:45] JAX Library
https://github.com/jax-ml/jax
[00:49:51] Llama 3
https://ai.meta.com/blog/meta-llama-3/
concept:
[00:24:10] Goodharts Law
https://en.wikipedia.org/wiki/Goodhart%27s_law
LINKS:
Full Transcript: https://app.rescript.info/share/04498ba49b081dbcc254c40ba7b56035
Download PDF transcript: https://app.rescript.info/api/public/sessions/8fe06defb6d2216d/pdf
Prof. Jakob Foerster
https://x.com/j_foerst
https://www.jakobfoerster.com/
University of Oxford Profile:
https://eng.ox.ac.uk/people/jakob-foerster/
REFS
[[00:00:05] ARC Benchmark, Chollet
https://github.com/fchollet/ARC-AGI
[00:09:45] JAX Library, Google Research
https://github.com/jax-ml/jax
[00:25:15] LLM ARChitect, Franzen et al.
https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf")
