How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, Ramp @aiDotEngineer
How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, Ramp  @aiDotEngineer
Uploaded November 2026 | Updated November 2026, 2 days ago
Ryan Stevens leads applied science at Ramp and explains how the team built end-to-end evals for Stack, its AI agent for bookkeepers. He starts by breaking an accountant's job into tasks like reconciliation and variance analysis, then builds synthetic businesses with full books so agents can be back-tested on the same data again and again. He covers accuracy versus pass@k, why the latest models still fail mostly on reasoning, and the experiments that didn't work, including auto-tuning prompts. Then he shows the ones that did: removing unneeded skills, testing memories across two time periods, and scheduled regression runs that report problems in Slack.

Chapters:
0:00 Intro
0:47 What's your agent's goal?
1:16 Breaking down a bookkeeper's job
2:43 Building a benchmark
3:20 Before and after you have users
4:26 Data is the bottleneck
4:48 Building synthetic businesses
5:45 Evals are just back-tests
7:08 How each task is graded
9:14 Accuracy vs. pass@k
10:24 Picking a model
11:18 Why models fail: reasoning
12:18 Experiments that failed
13:36 Skill ablation
15:01 Testing memories
17:01 Regression testing in the background
18:07 Takeaways

Ryan Stevens
ramp.com

Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!
How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, RampCan AI Understand a Distressed Credit? - Harneet Kaur, Axim IntelligenceThe One Thing Every AI Agent Framework Agrees On - Mislav Ivanda, DaytonaYour AI Isn’t the Problem. Your Infrastructure Is. — Jiachen Jiang, CoderWhy AI Startups Will Disappear Without Compute Markets — Evan Conrad, San Francisco ComputeYour AI Agents Shouldnt Care Where They Run - Nikhil Harithas, FactoryWhat Your Coding Agent Cant See in the Code - Tom Moor, LinearMCP = Mega Context Problem - Matt CareyWe Kept Rebuilding Our AI Agent. Heres What Survived - Fuad Ali, Arize AIThe Web Browser Is All You Need - Paul Klein IV, BrowserbaseYour Agent Just Paid for Nothing: 6 Attacks on x402 Payments - Ishween Kaur, SoFiTeach Your AI Agent to Build Products Everyone Can Use - Jemima Abu, CAIS
AI Engineer |

How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, Ramp

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER