Uploaded October 2025 | Updated September 2026, 2 weeks ago
We are joined by *Alex Shaw* and *Mike Merrill,* the creators of Terminal-Bench (tbench.ai/leaderboard). It is a coding agent benchmark that seemingly came out of nowhere to become the industry standard, adopted by all major frontier labs including Anthropic, OpenAI, and leading agent companies. They explain why they bet on terminal-based interaction over GUI computer use and how Nicholas Carlini at Anthropic became an early champion who helped get it featured prominently in Claude's model card. We also talked about the minimal agent "Terminus" designed to isolate model capabilities from agent optimizations.
00:00:00 Introduction and Terminal Bench Origins
00:03:57 The Anthropic Surprise and Industry Adoption
00:06:14 Terminal vs GUI: The Design Philosophy
00:07:53 Task Design and Benchmark Diversity
00:10:41 Example Deep Dive: Train FastText Task
00:15:50 Building a Meta-Benchmark Framework
00:21:03 Terminus Agent: Separating Model from Harness
00:25:34 The Future of Agent Evaluation
00:26:51 Roadmap: Cloud Hosting and Framework Evolution
00:29:18 Beyond Accuracy: Multi-Dimensional Evaluation
We are joined by *Alex Shaw* and *Mike Merrill,* the creators of Terminal-Bench (tbench.ai/leaderboard). It is a coding agent benchmark that seemingly came out of nowhere to become the industry standard, adopted by all major frontier labs including Anthropic, OpenAI, and leading agent companies. They explain why they bet on terminal-based interaction over GUI computer use and how Nicholas Carlini at Anthropic became an early champion who helped get it featured prominently in Claude's model card. We also talked about the minimal agent "Terminus" designed to isolate model capabilities from agent optimizations.
00:00:00 Introduction and Terminal Bench Origins
00:03:57 The Anthropic Surprise and Industry Adoption
00:06:14 Terminal vs GUI: The Design Philosophy
00:07:53 Task Design and Benchmark Diversity
00:10:41 Example Deep Dive: Train FastText Task
00:15:50 Building a Meta-Benchmark Framework
00:21:03 Terminus Agent: Separating Model from Harness
00:25:34 The Future of Agent Evaluation
00:26:51 Roadmap: Cloud Hosting and Framework Evolution
00:29:18 Beyond Accuracy: Multi-Dimensional Evaluation
![[State of Context Engineering] Agentic RAG, Context Rot, MCP, Subagents — Nina Lopatina, Contextual](https://i.ytimg.com/vi/tSRqTerZrH8/mqdefault.jpg "[State of Context Engineering] Agentic RAG, Context Rot, MCP, Subagents — Nina Lopatina, Contextual
From neuroscience PhD research on reward learning and decision making to building the infrastructure for *context engineering at scale,* *Nina Lopatina* has spent the last year watching a brand-new category emerge from prototype to production—and now shes leading the charge to turn context engineering from a collection of design patterns into a *full-stack discipline* with benchmarks, tooling, and real-world deployment at enterprise scale. We caught up with Nina live at *NeurIPS 2025* (her fifth!) to dig into the state of context engineering heading into 2026: why this year felt like *six months compressed into a year* (the category only really took hold in mid-2024), how *agentic RAG is now the baseline* (query reformulation into subqueries improved performance so dramatically it became the new standard), why *context rot is cited in every blog* but industry benchmarks at real scale (100k+ documents, billions of tokens) are still rare, how *MCP is both a driver and a flaw* for context engineering (giant JSON tool definitions stuff the context window, but MCP servers unlock rapid prototyping before you optimize down to direct API calls), the rise of *sub-agents with turn limits and explicit constraints* (unlimited agency degrades performance and causes hallucinations), why *instruction-following re-rankers* are critical for scaling retrieval across massive databases (more recall up front, more precision in the final context window), how *benchmarks are being saturated faster than ever* (Claude Code just saturated a Princeton benchmark released in October, with solutions so good the gold dataset had errors), the *KV cache decision-making framework* for multi-turn agents (stuff that doesnt change goes up front, stuff that changes a lot goes at the bottom), why shes *embodied-evaling frontier models as a snowboarding coach* (training for a 25-lap mogul race over 3–4 months, and why she had to close the window and restart because the model lost training context), and her thesis that 2026 will be the year context engineering moves from *component-level innovation to full-system design patterns*—where the conversation shifts from how do I optimize my re-ranker to what does the end-to-end architecture look like for reasoning over billions of tokens in production?
We discuss:
* What Contextual does: *end-to-end platform for context engineering across domains* (code, legal, retail, e-commerce, support), with multimodal ingestion, hybrid search, re-rankers, and dynamic agents
* The *first instruction-following re-ranker* (launched March 2024): latency is the biggest complaint, but for dynamic agents (where latency is less sensitive), its a game-changer for reasoning over large databases
* Why *agentic RAG is now the baseline:* query reformulation into subqueries improved performance so dramatically it became the new standard (normal RAG is dead)
* The *context engineering hackathon* (Retail Universe, ~100k documents, PDFs/CSVs/logs): Ninas team used a dynamic agent with turn limits and explicit constraints to avoid infinite sub-agent loops
* *Context rot:* everyone cites it, but Anthropics work putting numbers on it (e.g., at 700k tokens in a 1M context window, retrieval drops to 30%) is what made it actionable
* *Sub-agents with turn limits:* unlimited agency degrades performance and causes hallucinations, so explicit constraints (turn limits, validation loops) are critical for scale
* The need for *industry benchmarks at real scale:* most benchmarks use toy datasets, but the Retail Universe hackathon dataset (100k+ documents, billions of tokens) is closer to production reality
* *KV cache decision-making:* stuff that doesnt change (system prompt, early turns) goes up front, stuff that changes a lot (recent turns, dynamic context) goes at the bottom—critical for multi-turn agents
* Why *intentional context compression* matters: models arent great at compaction yet, so Nina proactively limits turns (even in Cursor, she opens a new window mid-conversation to avoid context loss)
—
Nina Lopatina
* Contextual AI: https://contextual.ai
* X: https://x.com/ninalopatina
* LinkedIn: https://linkedin.com/in/ninalopatina
00:00:00 Introduction: Nina Lopatina on Context Engineering at NeurIPS
00:04:34 The Death of Normal RAG: Rise of Agentic RAG and Query Reformulation
00:06:20 Sub-Agents and Turn Limits: Lessons from the Retail Universe Hackathon
00:09:07 Context Engineering in 2024: Design Patterns and the Prototyping Stage
00:10:17 Benchmarks and Scale: From Princeton HOW to Saturated Research Tasks
00:12:52 Context Rot, MCP, and Tool Selection Challenges
00:17:28 Prompt Optimization: Jeppa, ACE, and Evolutionary Approaches
00:19:42 KV Cache Strategy and Multi-Turn Agent Stability
00:22:30 Domain Generalization: Code, Legal, Retail, and Beyond
00:23:59 Predictions and Full System Design: The Future of Context Engineering")
