Uploaded August 2026 | Updated September 2026, 1 week ago
Jerry Li (University of Washington)
https://simons.berkeley.edu/talks/jerry-li-university-washington-2026-08-04
Diffusion Generative Modeling: Progress and Next Steps
A major bottleneck of standard auto-regressive large language models is that their inference process is inherently sequential, resulting in very long and costly inference times. To circumvent this, practitioners proposed a class of language models called diffusion language models, of which the masked diffusion model (MDM) is one of the most promising and successful. The MDM is able to sample out-of-order and, ostensibly, many tokens at once and in parallel. However, there is very limited rigorous understanding of how much parallel sampling these models can perform without noticeable degradation in their sampling performance. In this work, we give a new, exact characterization of the expected divergence between the true distribution and the sampled distribution, for any distribution and any unmasking schedule for the sampler, showing an elegant connection between MDM sampling and the classical theory of univariate function approximation.
Jerry Li (University of Washington)
https://simons.berkeley.edu/talks/jerry-li-university-washington-2026-08-04
Diffusion Generative Modeling: Progress and Next Steps
A major bottleneck of standard auto-regressive large language models is that their inference process is inherently sequential, resulting in very long and costly inference times. To circumvent this, practitioners proposed a class of language models called diffusion language models, of which the masked diffusion model (MDM) is one of the most promising and successful. The MDM is able to sample out-of-order and, ostensibly, many tokens at once and in parallel. However, there is very limited rigorous understanding of how much parallel sampling these models can perform without noticeable degradation in their sampling performance. In this work, we give a new, exact characterization of the expected divergence between the true distribution and the sampled distribution, for any distribution and any unmasking schedule for the sampler, showing an elegant connection between MDM sampling and the classical theory of univariate function approximation.
