Uploaded December 2025 | Updated September 2026, 2 weeks ago
_as with all demo-heavy and especially vision AI podcasts, we encourage watching along on our YouTube (and tossing us an upvote/subscribe if you like!)_
From SAM 1's 11-million-image data engine to SAM 2's memory-based video tracking, MSL’s Segment Anything project has redefined what's possible in computer vision. Now SAM 3 takes the next leap: *concept segmentation*—prompting with natural language like "yellow school bus" or "tablecloth" to detect, segment, and track _every_ instance across images and video, in real time, with human-level exhaustivity. And with the latest SAM Audio (https://x.com/aiatmeta/status/2000980784425931067?s=46), SAM can now even segment audio output!
We sat down with *Nikhila Ravi* (SAM lead at Meta) and *Pengchuan Zhang* (SAM 3 researcher) alongside *Joseph Nelson* (CEO, Roboflow) to unpack how SAM 3 unifies interactive segmentation, open-vocabulary detection, video tracking, and more into a single model that runs in 30ms on images and scales to real-time video on multi-GPU setups. We dig into the *data engine* that automated exhaustive annotation from two minutes per image down to 25 seconds using AI verifiers fine-tuned on Llama, the new *SACO (Segment Anything with Concepts)* benchmark with 200,000+ unique concepts vs. the previous 1.2k, how SAM 3 separates recognition from localization with a *presence token,* why decoupling the detector and tracker was critical to preserve object identity in video, how *SAM 3 Agents* unlock complex visual reasoning by pairing SAM 3 with multimodal LLMs like Gemini, and the real-world impact: 106 million smart polygons created on Roboflow saving humanity an estimated 130+ years of labeling time across fields from cancer research to underwater trash cleanup to autonomous vehicle perception.
We discuss:
* What *SAM 3* is: a unified model for concept-prompted segmentation, detection, and tracking in images and video using atomic visual concepts like "purple umbrella" or "watering can"
* How *concept prompts* work: short text phrases that find all instances of a category without manual clicks, plus visual exemplars (boxes, clicks) to refine and adapt on the fly
* Real-time performance: 30ms per image (100 detected objects on H200), 10 objects on 2×H200 video, 28 on 4×, 64 on 8×, with parallel inference and "fast mode" tracking
* The *SACO benchmark:* 200,000+ unique concepts vs. 1.2k in prior benchmarks, designed to capture the diversity of natural language and reach human-level exhaustivity
* The *data engine:* from 2 minutes per image (all-human) to 45 seconds (model-in-loop proposals) to 25 seconds (AI verifiers for mask quality and exhaustivity checks), fine-tuned on Llama 3.2
* Why *exhaustivity* is central: every instance must be found, verified by AI annotators, and manually corrected only when the model misses—automating the hardest part of segmentation at scale
* Architecture innovations: *presence token* to separate recognition ("is it in the image?") from localization ("where is it?"), decoupled detector and tracker to preserve identity-agnostic detection vs. identity-preserving tracking
* Building on Meta's ecosystem: Perception Encoder, DINO v2 detector, Llama for data annotation, and SAM 2's memory-based tracking backbone
* *SAM 3 Agents:* using SAM 3 as a visual tool for multimodal LLMs (Gemini, Llama) to solve complex visual reasoning tasks like "find the bigger character" or "what distinguishes male from female in this image"
* Fine-tuning with as few as 10 examples: domain adaptation for specialized use cases (Waymo vehicles, medical imaging, OCR-heavy scenes) and the outsized impact of negative examples
* Real-world impact at Roboflow: 106M smart polygons created, saving 130+ years of labeling time across cancer research, underwater trash cleanup, autonomous drones, industrial automation, and more
—
MSL FAIR team
* Nikhila: \
* Pengchuan: pzzhang.github.io/pzzhang
Joseph Nelson
* X: \
* LinkedIn: \
00:00:00 Introduction and the SAM Series Legacy
00:00:53 SAM 3 Launch: Three Models in One Release
00:05:30 Live Demo: Concept Prompting and Visual Exemplars
00:10:54 From Prototype to Production: The Evolution of Text Prompting
00:15:45 The Data Engine: Automating Exhaustive Annotation
00:14:10 Real-World Impact: 130 Years of Humanity Saved
00:25:11 Architecture Deep Dive: Decoupled Detection and Tracking
00:28:02 SAM 3 Agent: Bridging Vision and Language Models
00:33:20 Head-to-Head: SAM 3 vs Gemini and Florence
00:47:50 Video Understanding and the Masklet Detection Score
00:20:24 Fine-Tuning and Domain Adaptation: From Waymos to Medical Imaging
00:52:25 The Future of Perception: Native Vision vs Tool Calls
01:05:45 Building with SAM 3: Roboflow's Rapid Auto-Labeling
00:57:02 Open Source Philosophy and the Path to AGI
00:58:24 What's Next: SAM 4, Video Scale, and Beyond Human Performance
_as with all demo-heavy and especially vision AI podcasts, we encourage watching along on our YouTube (and tossing us an upvote/subscribe if you like!)_
From SAM 1's 11-million-image data engine to SAM 2's memory-based video tracking, MSL’s Segment Anything project has redefined what's possible in computer vision. Now SAM 3 takes the next leap: *concept segmentation*—prompting with natural language like "yellow school bus" or "tablecloth" to detect, segment, and track _every_ instance across images and video, in real time, with human-level exhaustivity. And with the latest SAM Audio (https://x.com/aiatmeta/status/2000980784425931067?s=46), SAM can now even segment audio output!
We sat down with *Nikhila Ravi* (SAM lead at Meta) and *Pengchuan Zhang* (SAM 3 researcher) alongside *Joseph Nelson* (CEO, Roboflow) to unpack how SAM 3 unifies interactive segmentation, open-vocabulary detection, video tracking, and more into a single model that runs in 30ms on images and scales to real-time video on multi-GPU setups. We dig into the *data engine* that automated exhaustive annotation from two minutes per image down to 25 seconds using AI verifiers fine-tuned on Llama, the new *SACO (Segment Anything with Concepts)* benchmark with 200,000+ unique concepts vs. the previous 1.2k, how SAM 3 separates recognition from localization with a *presence token,* why decoupling the detector and tracker was critical to preserve object identity in video, how *SAM 3 Agents* unlock complex visual reasoning by pairing SAM 3 with multimodal LLMs like Gemini, and the real-world impact: 106 million smart polygons created on Roboflow saving humanity an estimated 130+ years of labeling time across fields from cancer research to underwater trash cleanup to autonomous vehicle perception.
We discuss:
* What *SAM 3* is: a unified model for concept-prompted segmentation, detection, and tracking in images and video using atomic visual concepts like "purple umbrella" or "watering can"
* How *concept prompts* work: short text phrases that find all instances of a category without manual clicks, plus visual exemplars (boxes, clicks) to refine and adapt on the fly
* Real-time performance: 30ms per image (100 detected objects on H200), 10 objects on 2×H200 video, 28 on 4×, 64 on 8×, with parallel inference and "fast mode" tracking
* The *SACO benchmark:* 200,000+ unique concepts vs. 1.2k in prior benchmarks, designed to capture the diversity of natural language and reach human-level exhaustivity
* The *data engine:* from 2 minutes per image (all-human) to 45 seconds (model-in-loop proposals) to 25 seconds (AI verifiers for mask quality and exhaustivity checks), fine-tuned on Llama 3.2
* Why *exhaustivity* is central: every instance must be found, verified by AI annotators, and manually corrected only when the model misses—automating the hardest part of segmentation at scale
* Architecture innovations: *presence token* to separate recognition ("is it in the image?") from localization ("where is it?"), decoupled detector and tracker to preserve identity-agnostic detection vs. identity-preserving tracking
* Building on Meta's ecosystem: Perception Encoder, DINO v2 detector, Llama for data annotation, and SAM 2's memory-based tracking backbone
* *SAM 3 Agents:* using SAM 3 as a visual tool for multimodal LLMs (Gemini, Llama) to solve complex visual reasoning tasks like "find the bigger character" or "what distinguishes male from female in this image"
* Fine-tuning with as few as 10 examples: domain adaptation for specialized use cases (Waymo vehicles, medical imaging, OCR-heavy scenes) and the outsized impact of negative examples
* Real-world impact at Roboflow: 106M smart polygons created, saving 130+ years of labeling time across cancer research, underwater trash cleanup, autonomous drones, industrial automation, and more
—
MSL FAIR team
* Nikhila: \
* Pengchuan: pzzhang.github.io/pzzhang
Joseph Nelson
* X: \
* LinkedIn: \
00:00:00 Introduction and the SAM Series Legacy
00:00:53 SAM 3 Launch: Three Models in One Release
00:05:30 Live Demo: Concept Prompting and Visual Exemplars
00:10:54 From Prototype to Production: The Evolution of Text Prompting
00:15:45 The Data Engine: Automating Exhaustive Annotation
00:14:10 Real-World Impact: 130 Years of Humanity Saved
00:25:11 Architecture Deep Dive: Decoupled Detection and Tracking
00:28:02 SAM 3 Agent: Bridging Vision and Language Models
00:33:20 Head-to-Head: SAM 3 vs Gemini and Florence
00:47:50 Video Understanding and the Masklet Detection Score
00:20:24 Fine-Tuning and Domain Adaptation: From Waymos to Medical Imaging
00:52:25 The Future of Perception: Native Vision vs Tool Calls
01:05:45 Building with SAM 3: Roboflow's Rapid Auto-Labeling
00:57:02 Open Source Philosophy and the Path to AGI
00:58:24 What's Next: SAM 4, Video Scale, and Beyond Human Performance
![[State of Context Engineering] Agentic RAG, Context Rot, MCP, Subagents — Nina Lopatina, Contextual](https://i.ytimg.com/vi/tSRqTerZrH8/mqdefault.jpg "[State of Context Engineering] Agentic RAG, Context Rot, MCP, Subagents — Nina Lopatina, Contextual
From neuroscience PhD research on reward learning and decision making to building the infrastructure for *context engineering at scale,* *Nina Lopatina* has spent the last year watching a brand-new category emerge from prototype to production—and now shes leading the charge to turn context engineering from a collection of design patterns into a *full-stack discipline* with benchmarks, tooling, and real-world deployment at enterprise scale. We caught up with Nina live at *NeurIPS 2025* (her fifth!) to dig into the state of context engineering heading into 2026: why this year felt like *six months compressed into a year* (the category only really took hold in mid-2024), how *agentic RAG is now the baseline* (query reformulation into subqueries improved performance so dramatically it became the new standard), why *context rot is cited in every blog* but industry benchmarks at real scale (100k+ documents, billions of tokens) are still rare, how *MCP is both a driver and a flaw* for context engineering (giant JSON tool definitions stuff the context window, but MCP servers unlock rapid prototyping before you optimize down to direct API calls), the rise of *sub-agents with turn limits and explicit constraints* (unlimited agency degrades performance and causes hallucinations), why *instruction-following re-rankers* are critical for scaling retrieval across massive databases (more recall up front, more precision in the final context window), how *benchmarks are being saturated faster than ever* (Claude Code just saturated a Princeton benchmark released in October, with solutions so good the gold dataset had errors), the *KV cache decision-making framework* for multi-turn agents (stuff that doesnt change goes up front, stuff that changes a lot goes at the bottom), why shes *embodied-evaling frontier models as a snowboarding coach* (training for a 25-lap mogul race over 3–4 months, and why she had to close the window and restart because the model lost training context), and her thesis that 2026 will be the year context engineering moves from *component-level innovation to full-system design patterns*—where the conversation shifts from how do I optimize my re-ranker to what does the end-to-end architecture look like for reasoning over billions of tokens in production?
We discuss:
* What Contextual does: *end-to-end platform for context engineering across domains* (code, legal, retail, e-commerce, support), with multimodal ingestion, hybrid search, re-rankers, and dynamic agents
* The *first instruction-following re-ranker* (launched March 2024): latency is the biggest complaint, but for dynamic agents (where latency is less sensitive), its a game-changer for reasoning over large databases
* Why *agentic RAG is now the baseline:* query reformulation into subqueries improved performance so dramatically it became the new standard (normal RAG is dead)
* The *context engineering hackathon* (Retail Universe, ~100k documents, PDFs/CSVs/logs): Ninas team used a dynamic agent with turn limits and explicit constraints to avoid infinite sub-agent loops
* *Context rot:* everyone cites it, but Anthropics work putting numbers on it (e.g., at 700k tokens in a 1M context window, retrieval drops to 30%) is what made it actionable
* *Sub-agents with turn limits:* unlimited agency degrades performance and causes hallucinations, so explicit constraints (turn limits, validation loops) are critical for scale
* The need for *industry benchmarks at real scale:* most benchmarks use toy datasets, but the Retail Universe hackathon dataset (100k+ documents, billions of tokens) is closer to production reality
* *KV cache decision-making:* stuff that doesnt change (system prompt, early turns) goes up front, stuff that changes a lot (recent turns, dynamic context) goes at the bottom—critical for multi-turn agents
* Why *intentional context compression* matters: models arent great at compaction yet, so Nina proactively limits turns (even in Cursor, she opens a new window mid-conversation to avoid context loss)
—
Nina Lopatina
* Contextual AI: https://contextual.ai
* X: https://x.com/ninalopatina
* LinkedIn: https://linkedin.com/in/ninalopatina
00:00:00 Introduction: Nina Lopatina on Context Engineering at NeurIPS
00:04:34 The Death of Normal RAG: Rise of Agentic RAG and Query Reformulation
00:06:20 Sub-Agents and Turn Limits: Lessons from the Retail Universe Hackathon
00:09:07 Context Engineering in 2024: Design Patterns and the Prototyping Stage
00:10:17 Benchmarks and Scale: From Princeton HOW to Saturated Research Tasks
00:12:52 Context Rot, MCP, and Tool Selection Challenges
00:17:28 Prompt Optimization: Jeppa, ACE, and Evolutionary Approaches
00:19:42 KV Cache Strategy and Multi-Turn Agent Stability
00:22:30 Domain Generalization: Code, Legal, Retail, and Beyond
00:23:59 Predictions and Full System Design: The Future of Context Engineering")
