Uploaded November 2026 | Updated November 2026, 4 days ago
Cameron Wolfe, a staff research scientist at Netflix and author of the Deep (Learning) Focus newsletter, lays out a three-part framework for evaluating tasks with no single right answer, like creative writing. Human evaluation with calibrated guidelines is the North Star; LLM judges and rubrics, checked against human "golden" data, scale up offline evals; and A/B tests and production monitoring confirm results in the real world. Along the way he covers fighting noise with power analysis, error bars and resampling, and a Netflix project where an LLM judge's scores were causally linked to viewers abandoning shows.
Chapters:
0:00 Intro
0:50 Why evals are hard: noise
2:43 The spectrum of verifiable tasks
3:54 A three-part framework
4:27 Human evaluation as the North Star
5:19 Calibrating human reviewers
6:05 How much data, and what kind
8:10 Offline evals: LLM judges and rubrics
9:47 Checking the judge against humans
11:45 Fighting noise with statistics
13:17 Production monitoring and A/B tests
15:14 Keep improving the loop
15:45 Recap
Cameron Wolfe
netflix.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Cameron Wolfe, a staff research scientist at Netflix and author of the Deep (Learning) Focus newsletter, lays out a three-part framework for evaluating tasks with no single right answer, like creative writing. Human evaluation with calibrated guidelines is the North Star; LLM judges and rubrics, checked against human "golden" data, scale up offline evals; and A/B tests and production monitoring confirm results in the real world. Along the way he covers fighting noise with power analysis, error bars and resampling, and a Netflix project where an LLM judge's scores were causally linked to viewers abandoning shows.
Chapters:
0:00 Intro
0:50 Why evals are hard: noise
2:43 The spectrum of verifiable tasks
3:54 A three-part framework
4:27 Human evaluation as the North Star
5:19 Calibrating human reviewers
6:05 How much data, and what kind
8:10 Offline evals: LLM judges and rubrics
9:47 Checking the judge against humans
11:45 Fighting noise with statistics
13:17 Production monitoring and A/B tests
15:14 Keep improving the loop
15:45 Recap
Cameron Wolfe
netflix.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!

