Uploaded October 2026 | Updated November 2026, 4 days ago
A hands-on workshop from SallyAnn DeLucia, head of product at Arize AI, on moving from vibe checks to real evals. She covers what an eval is and how it differs from a unit test, the main types (LLM as a judge, decision models, code evals, annotations, agent as a judge), how to read a trace, how to write concrete pass/fail criteria, and how to build and iterate on an LLM judge in Arize's prompt playground.
Chapters
00:00 Intro and agenda
02:06 Why vibe checks don't scale
03:08 Is an eval just a unit test?
04:13 Why agents make evals harder
05:15 Types of evals
07:29 Workshop setup
10:16 Exploring the travel agent's traces
14:20 How to read a trace
19:03 Defining what good means
22:02 Writing pass/fail criteria
25:19 Building a dataset from traces
26:53 Building and testing an LLM judge
32:08 Next steps: alignment, online evals, Signal
SallyAnn DeLucia, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
A hands-on workshop from SallyAnn DeLucia, head of product at Arize AI, on moving from vibe checks to real evals. She covers what an eval is and how it differs from a unit test, the main types (LLM as a judge, decision models, code evals, annotations, agent as a judge), how to read a trace, how to write concrete pass/fail criteria, and how to build and iterate on an LLM judge in Arize's prompt playground.
Chapters
00:00 Intro and agenda
02:06 Why vibe checks don't scale
03:08 Is an eval just a unit test?
04:13 Why agents make evals harder
05:15 Types of evals
07:29 Workshop setup
10:16 Exploring the travel agent's traces
14:20 How to read a trace
19:03 Defining what good means
22:02 Writing pass/fail criteria
25:19 Building a dataset from traces
26:53 Building and testing an LLM judge
32:08 Next steps: alignment, online evals, Signal
SallyAnn DeLucia, Arize AI
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!










![AI Engineer NYC: OpenAI, Blackstone, Ramp, Coinbase, Apollo, Citadel, Basis, Runway, Compute Trading
Live from the Sheraton New York Times Square Hotel in New York City, AI Engineer New York 2026 is an AI conference for engineers and technical leaders building production AI across banking, hedge funds, fintech, insurance, and asset management.
Event: AI Engineer NYC
Dates: October 12–14, 2026
Venue: Sheraton New York Times Square Hotel
Chapters:
0:00 Livestream begins
14:59 Welcome Remarks — Ksenia Se (Turing Post)
20:29 Chasing capabilities while waiting for the frontier — Alexander Shevchenko (Ramp)
35:00 The Machine that Changed the World, vAI — Tariq Shaukat (Sonar)
52:04 Trusting Agents in Policy-Driven Workflows — Evan Kormos (Coinbase)
1:10:03 Building the Next Chapter of AI in Finance — Joseph Kim (OpenAI)
1:29:21 Security in the Age of AI: Recent Attacks, Incidents & Defenses — Andrey Rybka (Bloomberg)
1:59:37 Towards Autonomous Analysts in Finance — Yijia Liang (Compound AI)
2:25:00 Not All GPUs Are Deployed The Same — Oliver Kennon (SemiAnalysis)
2:49:46 How Compute Markets Work — Evan Conrad (San Francisco Compute Company)
3:15:09 The Dark Pools of Compute — Affaan Mustafa (Jump Trading) & Alejandro Soumah Fernandez (Jump Trading)
3:39:34 Compute Exchange — Carmen Li (Compute Exchange / Silicon Data)
3:58:42 Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar (Intuit Credit Karma) [Online]
4:28:27 The Agent Works. Does Anyone Trust It? — Michele Grant (Leading at Scale) [Online]
4:43:49 Tokens Are the Cheap Part — Steve Jamison (Jellyfish) [Online]
5:01:38 The Automated Security Master: keeping your master database clean with agents — Amol Kapoor (Nori Agentic) & Clifford Ressel (Nori Agentic) [Online] [Excerpt]
5:22:25 Built to Scale: Future-Proofing Your Technology Roadmap — John Stecher (Blackstone) & Scott Worland (Norm.ai)
5:49:35 Why Wall Street Still Doesn’t Trust AI — Vinoo Ganesh (Kepler)
6:11:48 Coding an Autonomous Wall Street Coworker: Custom Skills, Connectors, and Event Hooks — Philipp Schmid (Google DeepMind)
6:38:33 Your Company Has No Model of Itself: Bootstrapping a World Model from Slack, Decks and Meetings — Joao Wenzel (Citadel Securities)
7:05:28 Self-Improving Organizations — Will Brown (Prime Intellect)
7:26:05 The Cost of Accessibility in AI — Jemima Abu (CAIS) [Online]
7:36:29 Your AI Isn’t the Problem. Your Infrastructure Is. — Jiachen Jiang (Coder) [Online] [Excerpt]
7:44:56 Welcome Back — Featured Keynotes — Ksenia Se (Turing Post)
7:50:03 AI in Practice: An Inside Look at Apollos Approach — Michael Manapat (Rowspace) & Jack McDonald (Apollo Global Management)
8:09:22 Process Supervision for Agents That Do Long-Horizon Work — Mitchell Troyanovsky (Basis)
8:29:11 Everything will be Simulated — Cristóbal Valenzuela (Runway)
8:45:54 Closing Remarks — Ksenia Se (Turing Post) & swyx (AI Engineer)
Learn more about the event:
https://ai.engineer/nyc/2026#
Subscribe for more AI Engineer talks, livestreams, and event coverage:
/ @aidotengineer AI Engineer NYC: OpenAI, Blackstone, Ramp, Coinbase, Apollo, Citadel, Basis, Runway, Compute Trading](https://i.ytimg.com/vi/jiUC6IUMdvc/mqdefault.jpg)