Uploaded April 2025 | Updated September 2026, 2 weeks ago
Allen School Colloquia Series
Title: Pareto-efficient AI systems: Expanding the quality and efficiency frontier of AI
Speaker: Simran Arora (Stanford)
Date: April 1, 2025
Abstract: We have made exciting progress in AI by scaling massive models on massive amounts of data center compute. However, this represents a small fraction of AI’s potential. My work expands the Pareto frontier between the AI capabilities we can achieve and the long tail of compute constraints.
In this talk, we piece-by-piece build up to a language model architecture that expands the Pareto frontier between quality and throughput efficiency. The Transformer, AI’s current workhorse architecture, is memory hungry, limiting its throughput, or amount of data it can process per second. This has led to a Cambrian explosion of alternate architecture candidates proposed across prior work. Prior work paints an exciting picture: there are architectures that are asymptotically faster than the Transformer, while also matching its quality. However, I ask, if we’re using asymptotically faster building blocks, what if anything are we giving up in quality?
1. In part one, we understand the tradeoffs and show indeed, there’s no free lunch. I present my work to identify and explain the fundamental quality and efficiency tradeoffs between different classes of architectures. Methods I developed for this analysis are now
ubiquitous in the development of efficient language models.
2. In part two, we measure how existing architecture candidates fare along on the tradeoff space. While many proposed architectures are asymptotically fast, they are not wall-clock fast compared to the Transformer. I present ThunderKittens, a programming
library that I built to help AI researchers develop hardware-efficient AI algorithms.
3. In part three, we expand the Pareto frontier of the tradeoff space. I present the BASED architecture, which is built from simple, hardware-efficient components. In culmination, I released a suite of state-of-the-art 8B-405B parameter Transformer-free language models, per standard evaluations, all on an academic budget.
Given the massive investment into AI models, this work blending AI and systems has had significant impact and adoption in research, open-source, and industry.
Bio: Simran Arora is a PhD student at Stanford University advised by Chris Ré. Her research blends AI and systems towards expanding the Pareto frontier between AI capabilities and efficiency. Her machine learning research has appeared as Oral and Spotlight presentations at NeurIPS, ICML, and ICLR, including an Outstanding Paper award at NeurIPS and Best Paper award at ICML ES-FoMo. Her systems work has appeared at VLDB, SIGMOD, CIDR, and CHI, and her systems artifacts are widely used in research, open-source, and industry. In 2023, Simran created and taught the CS229s Systems for Machine Learning course at Stanford. She has also been supported by a SGF Sequoia Fellowship and the Stanford Computer Science Graduate Fellowship.
This video is closed captioned.
Allen School Colloquia Series
Title: Pareto-efficient AI systems: Expanding the quality and efficiency frontier of AI
Speaker: Simran Arora (Stanford)
Date: April 1, 2025
Abstract: We have made exciting progress in AI by scaling massive models on massive amounts of data center compute. However, this represents a small fraction of AI’s potential. My work expands the Pareto frontier between the AI capabilities we can achieve and the long tail of compute constraints.
In this talk, we piece-by-piece build up to a language model architecture that expands the Pareto frontier between quality and throughput efficiency. The Transformer, AI’s current workhorse architecture, is memory hungry, limiting its throughput, or amount of data it can process per second. This has led to a Cambrian explosion of alternate architecture candidates proposed across prior work. Prior work paints an exciting picture: there are architectures that are asymptotically faster than the Transformer, while also matching its quality. However, I ask, if we’re using asymptotically faster building blocks, what if anything are we giving up in quality?
1. In part one, we understand the tradeoffs and show indeed, there’s no free lunch. I present my work to identify and explain the fundamental quality and efficiency tradeoffs between different classes of architectures. Methods I developed for this analysis are now
ubiquitous in the development of efficient language models.
2. In part two, we measure how existing architecture candidates fare along on the tradeoff space. While many proposed architectures are asymptotically fast, they are not wall-clock fast compared to the Transformer. I present ThunderKittens, a programming
library that I built to help AI researchers develop hardware-efficient AI algorithms.
3. In part three, we expand the Pareto frontier of the tradeoff space. I present the BASED architecture, which is built from simple, hardware-efficient components. In culmination, I released a suite of state-of-the-art 8B-405B parameter Transformer-free language models, per standard evaluations, all on an academic budget.
Given the massive investment into AI models, this work blending AI and systems has had significant impact and adoption in research, open-source, and industry.
Bio: Simran Arora is a PhD student at Stanford University advised by Chris Ré. Her research blends AI and systems towards expanding the Pareto frontier between AI capabilities and efficiency. Her machine learning research has appeared as Oral and Spotlight presentations at NeurIPS, ICML, and ICLR, including an Outstanding Paper award at NeurIPS and Best Paper award at ICML ES-FoMo. Her systems work has appeared at VLDB, SIGMOD, CIDR, and CHI, and her systems artifacts are widely used in research, open-source, and industry. In 2023, Simran created and taught the CS229s Systems for Machine Learning course at Stanford. She has also been supported by a SGF Sequoia Fellowship and the Stanford Computer Science Graduate Fellowship.
This video is closed captioned.
![[Audio Descriptions] Faculty In Focus: Natasha Jaques](https://i.ytimg.com/vi/gqb-445BiTw/mqdefault.jpg "[Audio Descriptions] Faculty In Focus: Natasha Jaques
In the inaugural episode of the Allen School’s “Faculty in Focus” series, Assistant Professor Natasha Jaques describes her research in artificial intelligence aimed at building better AI agents. Jaques’ work draws from deep reinforcement learning and game theory to develop new approaches for ensuring that large language models like ChatGPT will be both effective and safe for users to interact with — regardless of the input.
For a version without audio descriptions, visit https://youtu.be/U2Xc3Ab_Has")
![[Audio Descriptions] I Am CSE: Ather Sharif](https://i.ytimg.com/vi/gtMoJAwW-Ms/mqdefault.jpg "[Audio Descriptions] I Am CSE: Ather Sharif
Allen School graduate student Ather Sharif of the UW’s ACE Lab and DUB Group describes his work on VoxLens, a tool that makes online data visualizations accessible to people who use screen readers, and explains why UW is the best place to do accessibility research.
This video is closed captioned.
A version of this video without audio descriptions is available here: https://youtu.be/oSBbE5PMKQs.")
![[Audio Descriptions] Faculty In Focus: Stephanie Wang](https://i.ytimg.com/vi/i6WuWgrUN5M/mqdefault.jpg "[Audio Descriptions] Faculty In Focus: Stephanie Wang
In this episode of the Allen School’s “Faculty in Focus” series, Assistant Professor Stephanie Wang talks about her research aimed at designing computer systems that can support advanced machine learning applications. The goal is to build infrastructure that is both scalable and future-proof — while also making it more accessible to a wider variety of people interested in developing and deploying state-of-the-art systems.
For a version without audio descriptions, visit https://youtu.be/6zgmbVEgm9o")
