Speech Recognition Is Not a Solved Problem — Pavan Kumar Reddy @MachineLearningStreetTalk
Speech Recognition Is Not a Solved Problem — Pavan Kumar Reddy  @MachineLearningStreetTalk
Uploaded September 2026 | Updated November 2026, 1 hour ago
Pavan Kumar Reddy leads audio research at Mistral AI. He joins Tim Scarfe for a deep technical tour of Voxtral — and explains why, after everything that has landed in the last two years, the frontier of deployed voice is still a cascade of specialised models rather than one end-to-end system.

IN PARTNERSHIP WITH MISTRAL AI:
---
This episode was produced in partnership with Mistral AI.
Mistral AI: mistral.ai
---

The conversation opens on architecture. Voxtral Chat feeds a 3B Ministral text trunk with continuous embeddings from an audio encoder, fed into the decoder as direct token input rather than through cross-attention as in Whisper, so the model can answer questions about emotion, timing and who spoke when without an intermediate transcript to lose them. The real-time model changes shape into a dual-stream decoder that consumes audio and emits text at the same time, at a configurable target delay down to 160ms, with slower streams running in parallel for anything that can afford to wait for more context.

On the generation side, Pavan explains why Voxtral TTS predicts continuous latents rather than discrete codec tokens, traces the lineage from SoundStream through EnCodec to Mimi's split of semantic and acoustic codebooks, and places finite scalar quantisation and flow matching in it. Tim presses on the engineering priors underneath: why a mel spectrogram instead of the raw waveform, what noise augmentation actually buys, and the point at which acoustic overfitting becomes somebody's fine-tuning problem. Then the failure modes. Speaker diarisation is emitted autoregressively as part of the transcript rather than by a separate head, which makes streaming diarisation fragile in a particular way — less context, late speaker changes, invented extra speakers. And because the architecture commits to what it has already predicted, a single out-of-distribution mistake can compound into looping or skipped segments, which is what DPO is there to correct: the negative supervision that pre-training and SFT cannot provide.

The last third is the argument Tim keeps returning to. Customers running voice agents over millions of sessions do not describe a solved problem, they describe scaffolding, with a sharp quality drop outside the top few languages. Cascades survive because each component stays separately adaptable, observable and constrainable — fine-tune the ASR for your acoustics, log what compliance demands, keep it on your own hardware. The interface has its own limit: voice alone is cognitive debt, because absorbing information and deciding in one serial audio stream is much harder than glancing at a menu. Voice becomes ubiquitous beside a screen, not instead of one.

---
TIMESTAMPS:
00:00:00 Cold open: the state changed
00:00:46 Why Mistral moved into audio
00:09:27 Inside Voxtral: the trunk, the encoder and dual streams
00:20:22 Speech that works in real time
00:30:52 How a voice becomes tokens
00:39:59 Flow matching, FSQ and the new codec
00:52:51 When speech models lose the speaker
01:03:23 Correcting hallucinations with preferences
01:12:12 Controlling synthetic speech
01:20:06 Why cascades still win
01:29:25 Speech in the wild
01:33:46 Audio models as interfaces
01:37:54 Why voice still needs a screen

---
REFERENCES:
paper:
[00:01:42] Mistral 7B
arxiv.org/abs/2310.06825
[00:09:38] Voxtral
arxiv.org/abs/2507.13264
[00:14:41] Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)
arxiv.org/abs/2212.04356
[00:19:11] Voxtral Realtime
arxiv.org/abs/2602.11298
[00:21:52] Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling
arxiv.org/abs/2509.08753
[00:30:52] Voxtral TTS
arxiv.org/abs/2603.25551
[00:32:38] SoundStream: An End-to-End Neural Audio Codec
arxiv.org/abs/2107.03312
[00:34:59] Flow Matching for Generative Modeling
arxiv.org/abs/2210.02747
[00:37:03] High Fidelity Neural Audio Compression (EnCodec)
arxiv.org/abs/2210.13438
[00:37:42] Moshi: a speech-text foundation model for real-time dialogue (Mimi)
arxiv.org/abs/2410.00037
[00:39:05] Finite Scalar Quantization: VQ-VAE Made Simple
arxiv.org/abs/2309.15505
[01:03:33] Direct Preference Optimization: Your Language Model is Secretly a Reward Model
arxiv.org/abs/2305.18290
benchmark:
[01:15:40] ElevenLabs v3 and Flash comparison in Voxtral TTS
arxiv.org/abs/2603.25551
dataset:
[00:46:14] Mozilla Common Voice datasets
commonvoice.mozilla.org/en/datasets
organization:
[00:01:24] Mistral AI
mistral.ai
[00:50:47] Hugging Face
huggingface.co
Speech Recognition Is Not a Solved Problem — Pavan Kumar ReddyExplosive AI Timeline Predictions [Gary Marcus, Daniel Kokotajlo, Dan Hendrycks]Are We Building Superintelligence Backwards? — Sara Saab & Enzo BlindowAIs which explore the worldAGI in 5 Years? Ben Goertzel on SuperintelligenceCan Outsourcing Thinking Make Us Dumber? [Prof. David Krakauer]AI HAS A BODY PROBLEM... [Dr. Maxwell Ramstead]One Step Closer to the Star Trek Voice AI Assistant!AI Can Look Safe and Still Be Dangerous | Daniel KokotajloAIs can now imagine video games in real-timeNeural Networks Are Elastic Origami! [Prof. Randall Balestriero]Compositionality - Prof. Kevin Ellis
Machine Learning Street Talk |

Speech Recognition Is Not a Solved Problem — Pavan Kumar Reddy

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER