Uploaded November 2025 | Updated September 2026, 2 weeks ago
At Ray Summit 2025, Mike Merrill from Stanford shares how the team is pushing the boundaries of agent evaluation by introducing Terminal-Bench—a hard, real-world–grounded benchmark designed to meaningfully measure progress toward autonomous, long-horizon AI agents.
He begins by outlining a key gap in today’s agent evaluation landscape: existing benchmarks either fail to reflect real-world tasks or are too easy to reliably differentiate the capabilities of frontier models. Terminal-Bench addresses this by providing a carefully curated set of challenging tasks that take place entirely within computer terminal environments—directly inspired by real workflows used by engineers and operators.
Mike then discusses insights gained from progress on Terminal-Bench so far, previewing what’s coming in Terminal-Bench 2.0, including expanded task sets, richer environment dynamics, and more nuanced evaluation metrics designed to stress-test reasoning, planning, and tool use.
Finally, he shares the team’s broader vision: unifying agent evaluation and training within a new open-source framework that enables reproducible, standardized, and scalable agent development.
Attendees will walk away with a deeper understanding of what it takes to evaluate real-world agentic capabilities—and how Terminal-Bench is shaping the future of agent benchmarks and AI research.
Liked this video? Check out other Ray Summit breakout session recordings youtube.com/playlist?list=PLzTswPQNepXllnU0C36WtkC0dqkAoDulh
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
At Ray Summit 2025, Mike Merrill from Stanford shares how the team is pushing the boundaries of agent evaluation by introducing Terminal-Bench—a hard, real-world–grounded benchmark designed to meaningfully measure progress toward autonomous, long-horizon AI agents.
He begins by outlining a key gap in today’s agent evaluation landscape: existing benchmarks either fail to reflect real-world tasks or are too easy to reliably differentiate the capabilities of frontier models. Terminal-Bench addresses this by providing a carefully curated set of challenging tasks that take place entirely within computer terminal environments—directly inspired by real workflows used by engineers and operators.
Mike then discusses insights gained from progress on Terminal-Bench so far, previewing what’s coming in Terminal-Bench 2.0, including expanded task sets, richer environment dynamics, and more nuanced evaluation metrics designed to stress-test reasoning, planning, and tool use.
Finally, he shares the team’s broader vision: unifying agent evaluation and training within a new open-source framework that enables reproducible, standardized, and scalable agent development.
Attendees will walk away with a deeper understanding of what it takes to evaluate real-world agentic capabilities—and how Terminal-Bench is shaping the future of agent benchmarks and AI research.
Liked this video? Check out other Ray Summit breakout session recordings youtube.com/playlist?list=PLzTswPQNepXllnU0C36WtkC0dqkAoDulh
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
