Uploaded February 2026 | Updated November 2026, 2 days ago
Joel Becker explains METR’s focus on Model Evaluation and Threat Research to assess whether AI could pose enormous or catastrophic risks. Becker discusses METR’s publicized work such as the time horizon chart (task difficulty measured in human-time at 50% reliability), how tasks are selected and constrained (economic relevance, auto-grading, non-messy scope), and why time horizon is often misinterpreted as how long agents run. They cover Opus 4.5’s perceived jump, challenges redoing developer productivity RCTs as workflows and adoption change, and why current models aren’t yet catastrophically dangerous while discontinuous capability gains remain possible if R&D loops fully automate. Becker also summarizes research linking compute growth slowdowns to slower capability progress, describes his Manifold trading story driven by a charity market he could influence, notes mixed social value of prediction markets, and previews METR’s 2026 plans, safeguards work, and hiring.
00:00 What METR Does
00:39 Podcast Intro
02:53 Threat Models Shift
03:33 Time Horizon Origin
04:56 Choosing Eval Tasks
06:25 Messy Real Work
08:10 HCAST And RE Bench
09:13 Human Time Misread
11:37 Opus 4.5 Surprise
14:27 Redoing Uplift RCTs
18:52 Measuring Productivity
20:55 Why Not Dangerous Yet
22:22 Capability Explosion
26:23 Benchmarks Miss Tail
28:08 Beyond One Number
29:50 Compute Slows Progress
30:47 Algorithms Need Compute
32:45 Lab Spend and Visibility
34:57 Cluster Timelines and Shipping
36:44 Prediction Markets and Models
38:10 Manifold Trading Story
39:52 Ethics and Insider Info
43:04 Beyond Benchmarks Evals
48:29 Harnesses and Scaffolding
51:39 METER Roadmap and Hiring
54:24 Karaoke and Human Voice
55:53 Closing Thoughts
Joel Becker explains METR’s focus on Model Evaluation and Threat Research to assess whether AI could pose enormous or catastrophic risks. Becker discusses METR’s publicized work such as the time horizon chart (task difficulty measured in human-time at 50% reliability), how tasks are selected and constrained (economic relevance, auto-grading, non-messy scope), and why time horizon is often misinterpreted as how long agents run. They cover Opus 4.5’s perceived jump, challenges redoing developer productivity RCTs as workflows and adoption change, and why current models aren’t yet catastrophically dangerous while discontinuous capability gains remain possible if R&D loops fully automate. Becker also summarizes research linking compute growth slowdowns to slower capability progress, describes his Manifold trading story driven by a charity market he could influence, notes mixed social value of prediction markets, and previews METR’s 2026 plans, safeguards work, and hiring.
00:00 What METR Does
00:39 Podcast Intro
02:53 Threat Models Shift
03:33 Time Horizon Origin
04:56 Choosing Eval Tasks
06:25 Messy Real Work
08:10 HCAST And RE Bench
09:13 Human Time Misread
11:37 Opus 4.5 Surprise
14:27 Redoing Uplift RCTs
18:52 Measuring Productivity
20:55 Why Not Dangerous Yet
22:22 Capability Explosion
26:23 Benchmarks Miss Tail
28:08 Beyond One Number
29:50 Compute Slows Progress
30:47 Algorithms Need Compute
32:45 Lab Spend and Visibility
34:57 Cluster Timelines and Shipping
36:44 Prediction Markets and Models
38:10 Manifold Trading Story
39:52 Ethics and Insider Info
43:04 Beyond Benchmarks Evals
48:29 Harnesses and Scaffolding
51:39 METER Roadmap and Hiring
54:24 Karaoke and Human Voice
55:53 Closing Thoughts







![Brex’s AI Hail Mary — With CTO James Reggio (acquired for $5B by Capital One!)
From building internal AI labs to becoming CTO of Brex, James Reggio has helped lead one of the most disciplined AI transformations inside a real financial institution where compliance, auditability, and customer trust actually matter.
We sat down with Reggio to unpack Brex’s three-pillar AI strategy (corporate, operational, and product AI) [https://www.brex.com/journal/brex-ai-native-operations], how SOP-driven agents beat overengineered RL in ops, why Brex lets employees “build their own AI stack” instead of picking winners [https://www.conductorone.com/customers/brex/], and how a small, founder-heavy AI team is shipping production agents to 40,000+ companies. Reggio also goes deep on Brex’s multi-agent “network” architecture, evals for multi-turn systems, agentic coding’s second-order effects on codebase understanding, and why the future of finance software looks less like dashboards and more like executive assistants coordinating specialist agents behind the scenes.
We discuss:
* Brex’s three-pillar AI strategy: corporate AI for 10x employee workflows, operational AI for cost and compliance leverage, and product AI that lets customers justify Brex as part of _their_ AI strategy to the board
* Why SOP-driven agents beat overengineered RL in finance ops, and how breaking work into auditable, repeatable steps unlocked faster automation in KYC, underwriting, fraud, and disputes
* Building an internal AI platform early: LLM gateways, prompt/version management, evals, cost observability, and why platform work quietly became the force multiplier behind everything else
* Multi-agent “networks” vs single-agent tools: why Brex’s EA-style assistant coordinates specialist agents (policy, travel, reimbursements) through multi-turn conversations instead of one-shot tool calls
* The audit agent pattern: separating detection, judgment, and follow-up into different agents to reduce false negatives without overwhelming finance teams
* Centralized AI teams without resentment: how Brex avoided “AI envy” by tying work to business impact and letting anyone transfer in if they cared deeply enough
* Letting employees build their own AI stack: ChatGPT vs Claude vs Gemini, Cursor vs Windsurf, and why Brex refuses to pick winners in fast-moving tool races
* Measuring adoption without vanity metrics: why “% of code written by AI” is the wrong KPI and what second-order effects (slop, drift, code ownership) actually matter
* Evals in the real world: regression tests from ops QA, LLM-as-judge for multi-turn agents, and why integration-style evals break faster than you expect
* Teaching AI fluency at scale: the user → advocate → builder → native framework, ops-led training, spot bonuses, and avoiding fear-based adoption
* Re-interviewing the entire engineering org: using agentic coding interviews internally to force hands-on skill upgrades without formal performance scoring
* Headcount in the age of agents: why Brex grew the business without growing engineering, and why AI amplifies bad architecture as fast as good decisions
* The future of finance software: why dashboards fade, assistants take over, and agent-to-agent collaboration becomes the real UI
—
James Reggio
* X: https://x.com/jamesreggio
* LinkedIn: https://www.linkedin.com/in/jamesreggio/
Where to find Latent Space
* X: https://x.com/latentspacepod
* Substack: https://www.latent.space/
00:00:00 Introduction
00:01:24 From Mobile Engineer to CTO: The Founders Path
00:03:00 Quitters Welcome: Building a Founder-Friendly Culture
00:05:13 The AI Team Structure: 10-Person Startup Within Brex
00:11:55 Building the Brex Agent Platform: Multi-Agent Networks
00:13:45 Tech Stack Decisions: TypeScript, Mastra, and MCP
00:24:32 Operational AI: Automating Underwriting, KYC, and Fraud
00:16:40 The Brex Assistant: Executive Assistant for Every Employee
00:40:26 Evaluation Strategy: From Simple SOPs to Multi-Turn Evals
00:37:11 Agentic Coding Adoption: Cursor, Windsurf, and the Engineering Interview
00:58:51 AI Fluency Levels: From User to Native
01:09:14 The Audit Agent Network: Finance Team Agents in Action
01:03:33 The Future of Engineering Headcount and AI Leverage Brex’s AI Hail Mary — With CTO James Reggio (acquired for $5B by Capital One!)](https://i.ytimg.com/vi/BKLvySNVBtM/mqdefault.jpg)


