Why Inference Matters 12x More Than Training | YC Paper Club @ycombinator
Why Inference Matters 12x More Than Training | YC Paper Club  @ycombinator
Uploaded October 2026 | Updated November 2026, 5 days ago
This week’s Paper Club is all about inference.

For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade. But inference is much more than calling model.forward. Every token has to move through a complicated stack of routing, caching, scheduling, kernels, memory, and hardware, all while balancing speed, cost, and quality.

We brought together researchers and builders from Together AI, Ollama, Luminal, and SGLang & RadixArk to take us through that stack from end to end. We’ll see how a token gets from an API request to a GPU and back, why the same open model can perform differently depending on who serves it, and how new approaches to compilers and caching are making inference faster and dramatically more efficient.

Join us for the next Paper Club on November 4th, where we'll talk AI for biology: events.ycombinator.com/yc-paper-club-nov-4

Chapters:
00:00 - Why inference matters more than training
08:24 - Three kinds of inference innovation
12:14 - Dan Fu: the lifecycle of an inference request
42:18 - Jeff Morgan: not all open model tokens are the same
1:04:31 - Joe Fioti: compiling models with search, not heuristics
1:30:30 - Liangsheng Yin: SGLang and memory for the agent era

Apply to Y Combinator: ycombinator.com/apply
Work at a startup: ycombinator.com/jobs
Why Inference Matters 12x More Than Training | YC Paper ClubThe Most AI-Pilled CEO We KnowAI-Native Discovery EnginesHow Legora Went From YC to $100M ARR in 18 MonthsWhy Robotics Still Isnt Solved - But Could Be Soon | YC Paper ClubLessons From Training Composer At Cursor And Building Meta/Nvidia Compute Clusters | YC Paper ClubWhy The Harness Matters More Than The Model | YC Paper ClubProving You’re HumanThe World’s Largest Electric Aircraft Just FlewMax Junestrand: You Need The Willingness To Learn Faster Than Anyone ElseZynga Founder: Consumer Is Not Investible Right Now - Thats Why You Should Build ItHow Jeetu Patel Runs Cisco Like the World’s Largest Startup
Y Combinator |

Why Inference Matters 12x More Than Training | YC Paper Club

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER