Recursive Language Models — Alex Zhang, MIT PhD @LatentSpacePod
Recursive Language Models — Alex Zhang, MIT PhD  @LatentSpacePod
Uploaded October 2026 | Updated November 2026, 3 days ago
From GPU kernels and KernelBench to Recursive Language Models, agent harnesses, and massive multi-agent swarms, Alex Zhang is exploring how much capability we’re leaving on the table by wrapping increasingly powerful models in primitive systems. In this episode, the MIT researcher behind RLMs joins swyx to unpack why Claude Code, Codex, and most coding agents are more similar than they look, why better harness design could unlock capabilities already latent in today’s models, and what happens when you can throw 10,000 agents and tens of millions of dollars at a single problem.

We go deep on GPU Mode and AI-written kernels, research taste and why academics should take bets industry labs won’t, GEV and alternatives to the standard autoregressive language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language model” of the future may actually be an invisible swarm of agents underneath a simple interface. We also discuss OpenAI’s massive agent experiments, Kimi swarms, open-ended research at Sakana AI, speculative programmatic tool calling, capability overhang, Neuralese, and where Alex thinks the next big research opportunities may lie.

We discuss:
• Why AI-generated GPU kernels still leave substantial room for human expertise
• How one expert insight can potentially replace enormous amounts of brute-force token search
• Why PhD students should take research bets that initially look trivial, weird, or pointless
• What SWE-bench, RLMs, ReAct, and Quiet-STaR reveal about research taste
• GEV and why a language model does not have to mean an autoregressive text-to-text decoder
• Why Claude Code, Codex, Pi, and many modern agent harnesses are structurally very similar
• How harness design can improve compositional generalization across tasks and domains
• RLMs: context offloading, code execution, recursive subagents, and shared memory
• Prime Agent, continual harnesses, and persistent agent-to-agent communication
• Why the model you query in the future may secretly be an entire swarm or scaffold
• OpenAI’s 10,000-agent experiment, 130B output tokens, and ~$40M-equivalent problem solving
• Why much of an agent swarm may be wasted search — and why convergence is still hard
• Kimi versus OpenAI’s approach to multi-agent systems
• Open-endedness, Sakana AI, and the problem of finding hidden gems in enormous amounts of generated work
• Why current frontier models may already have a large capability overhang
• Speculative programmatic tool calling and overlapping tool execution with generation
• Whether English, code, or an entirely new “Neuralese” constrains how models reason
• AI for science, fast-moving benchmarks, and how Alex chooses what research problems to bet on

Sponsorship and business inquiries: business@latent.space

—

Alex Zhang
• Website: alexzhang13.github.io
• X: @a1zhang

Timestamps
00:00:00 Hook
00:01:56 Introduction
00:02:45 GPU Mode, KernelBench, and AI-Written Kernels
00:09:34 Human Expertise vs. Brute-Force AI Search
00:15:16 Research Taste and Taking Big Bets
00:21:24 GEV and Rethinking the Language Model
00:30:59 Video Game Agents and the Harness Problem
00:32:57 Why Claude Code, Codex, and Pi Are So Similar
00:38:38 Harnesses as Compositional Generalizers
00:46:20 RLMs Explained
00:53:57 Prime Agent and Persistent Subagents
00:59:37 RLMs in the Wild
01:02:26 OpenAI Swarms and the Future of Language Models
01:09:22 Open-Endedness and Sakana AI
01:17:48 Kimi vs. OpenAI Agent Swarms
01:22:02 Capability Overhang and Speculative Tool Calling
01:30:15 Neuralese, Future Research, and AI for Science
Recursive Language Models — Alex Zhang, MIT PhD⚡️Jailbreaking AGI: Pliny the Liberator & John V on Red Teaming, BT6, and the Future of AI SecurityWhy Every Agent Needs a Box — Aaron Levie, BoxData Communication Complex Design Challenges Explained🔬Generating Molecules, Not Just Models⚡️Monty: the ultrafast Python interpreter by Agents for Agents — Samuel Colvin, PydanticMoonlake: Interactive, Multimodal World Models — with Chris Manning and Fan-yun SunThe Stove Guy: Sam DAmico Shows New AI Cooking Features on Americas Most Powerful Stove at ImpulseInside AI’s $10B+ Capital Flywheel — Martin Casado & Sarah Wang of a16z“We’re Investing Billions Into This One Thing”⚡️ 10x AI Engineers with $1m Salaries — Alex Lieberman & Arman Hezarkhani, TenexThe Agents Economy Backbone - with Emily Glassberg Sands, Head of Data & AI at Stripe
Latent Space |

Recursive Language Models — Alex Zhang, MIT PhD

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER