Uploaded October 2026 | Updated November 2026, 3 days ago
Which GPU clouds are worth trusting with your AI workloads?
This Lightning podcast episode explores SemiAnalysis’s ClusterMAX 3.0 and the questions that matter when evaluating managed GPU clusters.
About the research: ClusterMAX evaluates configuration, workload performance, reliability and response to failures through hands-on testing, documentation review and customer feedback.
Explore the research:
ClusterMAX 3.0 report: newsletter.semianalysis.com/p/clustermax-30-the-industry-standard
Rankings: clustermax.ai/v3
Methodology: clustermax.ai/overview
Evaluation criteria: clustermax.ai/criteria
Open-source audit tool: github.com/SemiAnalysisAI/ClusterMAX
Sponsorship and business inquiries: business@latent.space
#ClusterMAX #SemiAnalysis #AIInfrastructure
Which GPU clouds are worth trusting with your AI workloads?
This Lightning podcast episode explores SemiAnalysis’s ClusterMAX 3.0 and the questions that matter when evaluating managed GPU clusters.
About the research: ClusterMAX evaluates configuration, workload performance, reliability and response to failures through hands-on testing, documentation review and customer feedback.
Explore the research:
ClusterMAX 3.0 report: newsletter.semianalysis.com/p/clustermax-30-the-industry-standard
Rankings: clustermax.ai/v3
Methodology: clustermax.ai/overview
Evaluation criteria: clustermax.ai/criteria
Open-source audit tool: github.com/SemiAnalysisAI/ClusterMAX
Sponsorship and business inquiries: business@latent.space
#ClusterMAX #SemiAnalysis #AIInfrastructure

![[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang
From creating *SWE-bench* in a Princeton basement to shipping *CodeClash,* *SWE-bench Multimodal,* and *SWE-bench Multilingual,* *John Yang* has spent the last year and a half watching his benchmark become the de facto standard for evaluating AI coding agents—trusted by Cognition (Devin), OpenAI, Anthropic, and every major lab racing to solve software engineering at scale. We caught up with John live at *NeurIPS 2025* to dig into the state of code evals heading into 2026: why *SWE-bench went from ignored (October 2023) to the industry standard* after Devins launch (and how Walden emailed him two weeks before the big reveal), how the benchmark evolved from Django-heavy to *nine languages across 40 repos* (JavaScript, Rust, Java, C, Ruby), why *unit tests as verification are limiting* and long-running agent tournaments might be the future (CodeClash: agents maintain codebases, compete in arenas, and iterate over multiple rounds), the *proliferation of SWE-bench variants* (SWE-bench Pro, SWE-bench Live, SWE-Efficiency, AlgoTune, SciCode) and how benchmark authors are now justifying their splits with curation techniques instead of just more repos, why *Tau-benchs impossible tasks controversy* is actually a feature not a bug (intentionally including impossible tasks flags cheating), the tension between *long autonomy (5-hour runs) vs. interactivity* (Cognitions emphasis on fast back-and-forth), how *Terminal-bench unlocked creativity* by letting PhD students and non-coders design environments beyond GitHub issues and PRs, the *academic data problem* (companies like Cognition and Cursor have rich user interaction data, academics need user simulators or compelling products like LMArena to get similar signal), and his vision for *CodeClash as a testbed for human-AI collaboration*—freeze model capability, vary the collaboration setup (solo agent, multi-agent, human+agent), and measure how interaction patterns change as models climb the ladder from code completion to full codebase reasoning.
We discuss:
* Johns path: *Princeton → SWE-bench (October 2023) → Stanford PhD with Diyi Yang and the Iris Group,* focusing on code evals, human-AI collaboration, and long-running agent benchmarks
* The *SWE-bench origin story:* released October 2023, mostly ignored until *Cognitions Devin launch* kicked off the arms race (Walden emailed John two weeks before: we have a good number)
* *SWE-bench Verified:* the curated, high-quality split that became the standard for serious evals
* *SWE-bench Multimodal and Multilingual:* nine languages (JavaScript, Rust, Java, C, Ruby) across 40 repos, moving beyond the Django-heavy original distribution
* The *SWE-bench Pro controversy:* independent authors used the SWE-bench name without Johns blessing, but hes okay with it (congrats to them, its a great benchmark)
* *CodeClash:* Johns new benchmark for *long-horizon development*—agents maintain their own codebases, edit and improve them each round, then compete in arenas (programming games like Halite, economic tasks like GDP optimization)
* *SWE-Efficiency* (Jeffrey Maugh, Johns high school classmate): optimize code for speed without changing behavior (parallelization, SIMD operations)
* *AlgoTune, SciCode, Terminal-bench, Tau-bench, SecBench, SRE-bench:* the Cambrian explosion of code evals, each diving into different domains (security, SRE, science, user simulation)
* The *Tau-bench impossible tasks debate:* some tasks are underspecified or impossible, but John thinks thats actually a feature (flags cheating if you score above 75%)
* *Cognitions research focus:* codebase understanding (retrieval++), helping humans understand their own codebases, and automatic context engineering for LLMs (research sub-agents)
* The vision: *CodeClash as a testbed for human-AI collaboration*—vary the setup (solo agent, multi-agent, human+agent), freeze model capability, and measure how interaction changes as models improve
—
John Yang
* SWE-bench: https://www.swebench.com
* X: https://x.com/jyangballin
00:00:00 Introduction: John Yang on SWE-bench and Code Evaluations
00:00:31 SWE-bench Origins and Devons Impact on the Coding Agent Arms Race
00:01:09 SWE-bench Ecosystem: Verified, Pro, Multimodal, and Multilingual Variants
00:02:17 Moving Beyond Django: Diversifying Code Evaluation Repositories
00:03:08 Code Clash: Long-Horizon Development Through Programming Tournaments
00:04:41 From Halite to Economic Value: Designing Competitive Coding Arenas
00:06:04 Ofirs Lab: SWE-ficiency, AlgoTune, and SciCode for Scientific Computing
00:07:52 The Benchmark Landscape: TAU-bench, Terminal-bench, and User Simulation
00:09:20 The Impossible Task Debate: Refusals, Ambiguity, and Benchmark Integrity
00:12:32 The Future of Code Evals: Long Autonomy vs Human-AI Collaboration
00:14:37 Call to Action: User Interaction Data and Codebase Understanding Research [State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang](https://i.ytimg.com/vi/MxB-xRGXxkk/mqdefault.jpg)
![[State of Evals] LMArenas $1.7B Vision — Anastasios Angelopoulos, LMArena
_We are reupping this episode after LMArena announced their fresh Series A (_https://www.theinformation.com/articles/ai-evaluation-startup-lmarena-valued-1-7-billion-new-funding-round?rc=luxwz4_), raising $150m at a $1.7B valuation, with $30M annualized consumption revenue (aka $2.5m MRR) after their September evals product launch._
—-
From building *LMArena* in a Berkeley basement to raising $100M and becoming the *de facto leaderboard for frontier AI,* *Anastasios Angelopoulos* returns to Latent Space to recap 2025 in one of the most influential platforms in AI—trusted by millions of users, every major lab, and the entire industry to answer one question: *which model is actually best for real-world use cases?* We caught up with Anastasios live at *NeurIPS 2025* to dig into the origin story (spoiler: it started as an academic project incubated by *Anjney Midha at a16z,* who formed an entity and gave grants before they even committed to starting a company), why they decided to spin out instead of staying academic or nonprofit (the only way to scale was to build a company), how theyre spending that $100M (inference costs, React migration off Gradio, and hiring world-class talent across ML, product, and go-to-market), the *leaderboard delusion controversy* and why their response demolished the papers claims (factual errors, misrepresentation of open vs. closed source sampling, and ignoring the transparency of preview testing that the community loves), why *platform integrity comes first* (the public leaderboard is a charity, not a pay-to-play system—models cant pay to get on, cant pay to get off, and scores reflect millions of real votes), how theyre expanding into *occupational verticals* (medicine, legal, finance, creative marketing) and *multimodal arenas* (video coming soon), why *consumer retention is earned every single day* (sign-in and persistent history were the unlock, but users are fickle and can leave at any moment), and his vision for Arena as the *central evaluation platform* that provides the North Star for the industry—constantly fresh, immune to overfitting, and grounded in millions of real-world conversations from real users.
We discuss:
* The *$100M raise:* use of funds is primarily *inference costs* (funding free usage for tens of millions of monthly conversations), *React migration off Gradio* (custom loading icons, better developer hiring, more flexibility), and hiring world-class talent
* The scale: *250M+ conversations* on the platform, tens of millions per month, 25% of users do software for a living, and half of users are now logged in
* The *leaderboard illusion controversy:* Cohere researchers claimed undisclosed private testing created inequities, but Arenas response demolished the papers factual errors (misrepresented open vs. closed source sampling, ignored transparency of preview testing that the community loves)
* Why *preview testing is loved by the community:* secret codenames (Gemini Nano Banana, named after PM Nainas nickname), early access to unreleased models, and the thrill of being first to vote on frontier capabilities
* The *Nano Banana moment:* changed Googles market share overnight, billions of dollars in stock movement, and validated that multimodal models (image generation, video) are economically critical for marketing, design, and AI-for-science
* New categories: *occupational and expert arenas* (medicine, legal, finance, creative marketing), *Code Arena,* and *video arena* coming soon
00:00:00 Introduction: Anastasios from Arena and the LM Arena Journey
00:01:36 The Anjney Midha Incubation: From Berkeley Basement to Startup
00:02:47 The Decision to Start a Company: Scaling Beyond Academia
00:03:38 The $100M Raise: Use of Funds and Platform Economics
00:05:10 Arenas User Base: 5M+ Users and Diverse Demographics
00:06:02 The Competitive Landscape: Artificial Analysis, AI.xyz, and Arenas Differentiation
00:08:12 Educational Value and Learning from the Community
00:08:41 Technical Migration: From Gradio to React and Platform Evolution
00:10:18 Leaderboard Delusion Paper: Addressing Critiques and Maintaining Integrity
00:12:29 Nano Banana Moment: How Preview Models Create Market Impact
00:13:41 Multimodal AI and Image Generation: From Skepticism to Economic Value
00:15:37 Core Principles: Platform Integrity and the Public Leaderboard as Charity
00:18:29 Future Roadmap: Expert Categories, Multimodal, Video, and Occupational Verticals
00:19:10 API Strategy and Focus: Doing One Thing Well
00:19:51 Community Management and Retention: Sign-In, History, and Daily Value
00:22:21 Partnerships and Agent Evaluation: From Devon to Full-Featured Harnesses
00:21:49 Hiring and Building a High-Performance Team [State of Evals] LMArenas $1.7B Vision — Anastasios Angelopoulos, LMArena](https://i.ytimg.com/vi/NBnOk0Uy9ig/mqdefault.jpg)







