Uploaded November 2026 | Updated November 2026, 2 days ago
Ryan Stevens leads applied science at Ramp and explains how the team built end-to-end evals for Stack, its AI agent for bookkeepers. He starts by breaking an accountant's job into tasks like reconciliation and variance analysis, then builds synthetic businesses with full books so agents can be back-tested on the same data again and again. He covers accuracy versus pass@k, why the latest models still fail mostly on reasoning, and the experiments that didn't work, including auto-tuning prompts. Then he shows the ones that did: removing unneeded skills, testing memories across two time periods, and scheduled regression runs that report problems in Slack.
Chapters:
0:00 Intro
0:47 What's your agent's goal?
1:16 Breaking down a bookkeeper's job
2:43 Building a benchmark
3:20 Before and after you have users
4:26 Data is the bottleneck
4:48 Building synthetic businesses
5:45 Evals are just back-tests
7:08 How each task is graded
9:14 Accuracy vs. pass@k
10:24 Picking a model
11:18 Why models fail: reasoning
12:18 Experiments that failed
13:36 Skill ablation
15:01 Testing memories
17:01 Regression testing in the background
18:07 Takeaways
Ryan Stevens
ramp.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
Ryan Stevens leads applied science at Ramp and explains how the team built end-to-end evals for Stack, its AI agent for bookkeepers. He starts by breaking an accountant's job into tasks like reconciliation and variance analysis, then builds synthetic businesses with full books so agents can be back-tested on the same data again and again. He covers accuracy versus pass@k, why the latest models still fail mostly on reasoning, and the experiments that didn't work, including auto-tuning prompts. Then he shows the ones that did: removing unneeded skills, testing memories across two time periods, and scheduled regression runs that report problems in Slack.
Chapters:
0:00 Intro
0:47 What's your agent's goal?
1:16 Breaking down a bookkeeper's job
2:43 Building a benchmark
3:20 Before and after you have users
4:26 Data is the bottleneck
4:48 Building synthetic businesses
5:45 Evals are just back-tests
7:08 How each task is graded
9:14 Accuracy vs. pass@k
10:24 Picking a model
11:18 Why models fail: reasoning
12:18 Experiments that failed
13:36 Skill ablation
15:01 Testing memories
17:01 Regression testing in the background
18:07 Takeaways
Ryan Stevens
ramp.com
Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!










