Ship Fast, Break Nothing: How AI Tests What AI Builds | Sophie Zeng & Sreeja Thummala from Meta @scaleconference
Ship Fast, Break Nothing: How AI Tests What AI Builds | Sophie Zeng & Sreeja Thummala from Meta  @scaleconference
Uploaded October 2026 | Updated November 2026, 2 days ago
Meta engineers Suja and Sophie share how they've rebuilt software testing for the AI era, moving beyond traditional "landed tests" that only catch known bugs to a new system of disposable, AI-generated tests that target specific code changes.

When a risky change appears, their system scores diffs for risk factors like blast radius and intent mismatch, then generates property-based or mutation-based tests designed to fail only when that specific change introduces defects [6:09] [6:45]. Context is critical: providing the generator with code change details and author intent roughly doubled catch rates at each step.

A key innovation is the two-stage architecture: an aggressive generator proposes many hypotheses, while an LLM judge filters out intended behavioral changes to protect engineer attention, discarding the vast majority of generated tests before they reach humans.This reduced engineer time spent chasing bugs by 70% despite not finding fewer actual bugs.

The system has prevented 74 documented safety incidents and bent privacy incident growth to one-fifth of forecasted levels through counterfactual measurement .

The talk closes with three ongoing shifts: tests becoming disposable, solving the long-term trust challenge, and moving testing left from review to co-authorship during code writing.
Ship Fast, Break Nothing: How AI Tests What AI Builds | Sophie Zeng & Sreeja Thummala from MetaLess Problems, More Building!Live from SCCC: MetaRoCE: From Spec to NIC to Open Source | Balakrishnan Raman and Sandeep NagarajEvolving GenAI Media Infrastructure Deployments | Rushaan Mahajan, Sima LabsOur Journey to Safely Unleash Agents at Meta Scale | David Pariag from MetaAdopting AV1 for Real-Time Communication (RTC) at Scale | Yu Chen Sun Eric from MetaLive from SCCC: MetaRoCE: Meta’s RDMA Transport | Arvind Srinivasan and Kingshuk MandalInside Instagrams VR Revolution: AI-Powered 3D Content at ScaleWorld Models for Robotics from Reactor | Alberto Taiuti from Reactor🚀 Registration is LIVE!Co-Designing Communication for AI Accelerators | Rajeev Nair and Wes Bland from MetaScaling Media AI: From Pixel-Level Enhancement to Actionable Intelligence on Intel® Xeon® by Intel
@Scale |

Ship Fast, Break Nothing: How AI Tests What AI Builds | Sophie Zeng & Sreeja Thummala from Meta

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER