How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI @aiDotEngineer
How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI  @aiDotEngineer
Uploaded October 2026 | Updated November 2026, 4 days ago
A hands-on workshop from SallyAnn DeLucia, head of product at Arize AI, on moving from vibe checks to real evals. She covers what an eval is and how it differs from a unit test, the main types (LLM as a judge, decision models, code evals, annotations, agent as a judge), how to read a trace, how to write concrete pass/fail criteria, and how to build and iterate on an LLM judge in Arize's prompt playground.

Chapters
00:00 Intro and agenda
02:06 Why vibe checks don't scale
03:08 Is an eval just a unit test?
04:13 Why agents make evals harder
05:15 Types of evals
07:29 Workshop setup
10:16 Exploring the travel agent's traces
14:20 How to read a trace
19:03 Defining what good means
22:02 Writing pass/fail criteria
25:19 Building a dataset from traces
26:53 Building and testing an LLM judge
32:08 Next steps: alignment, online evals, Signal

SallyAnn DeLucia, Arize AI

Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
How to Write Your First AI Eval - SallyAnn DeLucia, Arize AIWhy Are We Still Governing AI Agents Like Software? - Tim Tyler, GraviteeYour Agent Doesnt Need an LLM for Every Decision - Raymond Weitekamp, OpenProseDesigning Voice Agents for Real Conversations - Chintan Agrawal & Daniel Wirjo, AWSAI Writes Code Faster Than Uber Can Merge It - Preetam Dwivedi & Manjari Akella, UberWhy Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, GooglePick Problems That Survive When AI Solves Coding - Ara Khan, ClineRealtime multiplayer, automation, and you! — Idan Gazit, GitHubWhat a $900M Bank Error Teaches Us About AI Agents - John Feminella, Two SigmaBeyond Static Intelligence: Evaluating Continual Learning — Parth Asawa, UC BerkeleyYour Company Has No Model of Itself. Agents Can Build One - Joao Wenzel, Citadel SecuritiesAI Engineer NYC: OpenAI, Blackstone, Ramp, Coinbase, Apollo, Citadel, Basis, Runway, Compute Trading
AI Engineer |

How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER