Uploaded May 2026 | Updated September 2026, 1 week ago
Peter Zhong is on the AI team at Replit. He walked up to the SaaStr floor as a surprise guest on Day 1 to talk about the work that happens every time a new AI model drops, why evaluation matters more than chasing the latest release, how Replit just open-sourced its eval framework (ViBench), and why specialized agents (like the Security Agent) are the right answer for high-stakes use cases.
In this interview we cover:
• The work behind every new model release: benchmark, evaluate, test against real user workloads
• Cost, quality, intelligence: optimizing for the best user experience, not the most expensive model
• ViBench: Replit's newly open-sourced eval framework for vibe coding
• Specialized agents like the Security Agent — and why security gets its own model
• Security woven through code review + optimization, not just bolted on
• Peter's personal AI workflow: human-driven planning, agent execution
• SOTA hype vs. ground truth: evolutionary vs. revolutionary model releases
• When the "design mode" launch was the real wow moment
• Why slides and animations turned out to be the killer use case for many users
• Vibe coding going mainstream — and what that means
• Replit becoming "customized mission control" for your business
• "Distribution is all you need": the next hard problem after building
Parent stream replay (full Day 1): youtu.be/EWv2cIQpYbo
Timestamps:
00:00 Peter joins (surprise walk-up)
00:09 What it's like working at Replit after academia
00:38 Stop worrying about the latest model release
01:09 The work behind every new model: extensive benchmarking
01:31 Cost, quality, intelligence: not just intelligence
01:40 Why all models aren't equal (design, context, orchestration)
02:24 The secret sauce: evaluation
02:38 ViBench: Replit's open-source eval framework
03:06 Specialized agents: the Security Agent example
03:30 Why Replit uses a different model for security
04:00 Security woven through code review + optimization
04:30 Supply chain attacks + sleeping soundly with Replit
04:58 How Peter keeps up with LLM releases
05:11 The "vibe eval" perspective
05:30 A/B testing + human feedback on designs
06:10 Peter's personal AI workflow: human-driven planning
07:00 SOTA hype vs. ground truth
07:23 Evolutionary vs. revolutionary model releases
07:43 The design mode launch as a real "wow" moment
08:06 Internal experimentation + Replit's hacky culture
08:43 Slides + videos as unexpected use cases
09:06 Vibe coding going mainstream
09:28 Building the whole business: web app, mobile, slides, marketing
09:38 "Creativity actually unlocked"
10:23 Replit expanding into full product launch workflows
11:02 "Customized mission control"
11:20 Building isn't the hard part anymore. Distribution is.
11:57 "Attention is all you need" → "Distribution is all you need"
12:44 Peter's wrap-up: proud to see builders at work
Subscribe for more: youtube.com/@replit
Follow Manny: https://x.com/MannyBernabe
Follow Raymmar: https://x.com/raymmar
Follow Peter: https://x.com/Lambda_freak
Check out ViBench: vibench.ai
Michele's Code with Claude talk on ViBench: youtu.be/snroDwX1-JU
#Replit #AI #VibeCoding #AIAgents #ViBench #ModelEvaluation #SaaStr #FutureOfWork #AINative #B2BAI
Peter Zhong is on the AI team at Replit. He walked up to the SaaStr floor as a surprise guest on Day 1 to talk about the work that happens every time a new AI model drops, why evaluation matters more than chasing the latest release, how Replit just open-sourced its eval framework (ViBench), and why specialized agents (like the Security Agent) are the right answer for high-stakes use cases.
In this interview we cover:
• The work behind every new model release: benchmark, evaluate, test against real user workloads
• Cost, quality, intelligence: optimizing for the best user experience, not the most expensive model
• ViBench: Replit's newly open-sourced eval framework for vibe coding
• Specialized agents like the Security Agent — and why security gets its own model
• Security woven through code review + optimization, not just bolted on
• Peter's personal AI workflow: human-driven planning, agent execution
• SOTA hype vs. ground truth: evolutionary vs. revolutionary model releases
• When the "design mode" launch was the real wow moment
• Why slides and animations turned out to be the killer use case for many users
• Vibe coding going mainstream — and what that means
• Replit becoming "customized mission control" for your business
• "Distribution is all you need": the next hard problem after building
Parent stream replay (full Day 1): youtu.be/EWv2cIQpYbo
Timestamps:
00:00 Peter joins (surprise walk-up)
00:09 What it's like working at Replit after academia
00:38 Stop worrying about the latest model release
01:09 The work behind every new model: extensive benchmarking
01:31 Cost, quality, intelligence: not just intelligence
01:40 Why all models aren't equal (design, context, orchestration)
02:24 The secret sauce: evaluation
02:38 ViBench: Replit's open-source eval framework
03:06 Specialized agents: the Security Agent example
03:30 Why Replit uses a different model for security
04:00 Security woven through code review + optimization
04:30 Supply chain attacks + sleeping soundly with Replit
04:58 How Peter keeps up with LLM releases
05:11 The "vibe eval" perspective
05:30 A/B testing + human feedback on designs
06:10 Peter's personal AI workflow: human-driven planning
07:00 SOTA hype vs. ground truth
07:23 Evolutionary vs. revolutionary model releases
07:43 The design mode launch as a real "wow" moment
08:06 Internal experimentation + Replit's hacky culture
08:43 Slides + videos as unexpected use cases
09:06 Vibe coding going mainstream
09:28 Building the whole business: web app, mobile, slides, marketing
09:38 "Creativity actually unlocked"
10:23 Replit expanding into full product launch workflows
11:02 "Customized mission control"
11:20 Building isn't the hard part anymore. Distribution is.
11:57 "Attention is all you need" → "Distribution is all you need"
12:44 Peter's wrap-up: proud to see builders at work
Subscribe for more: youtube.com/@replit
Follow Manny: https://x.com/MannyBernabe
Follow Raymmar: https://x.com/raymmar
Follow Peter: https://x.com/Lambda_freak
Check out ViBench: vibench.ai
Michele's Code with Claude talk on ViBench: youtu.be/snroDwX1-JU
#Replit #AI #VibeCoding #AIAgents #ViBench #ModelEvaluation #SaaStr #FutureOfWork #AINative #B2BAI
