LSTM: The Comeback Story? [Prof. Sepp Hochreiter] @MachineLearningStreetTalk
LSTM: The Comeback Story? [Prof. Sepp Hochreiter]  @MachineLearningStreetTalk
Uploaded February 2025 | Updated November 2026, 4 days ago
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!
centml.ai/pricing

Sepp Hochreiter, the inventor of LSTM networks, makes a forceful case that large language models are fundamentally limited -- they store and retrieve human knowledge rather than reason about it. He walks through the technical history from the original 1991 vanishing gradient problem through to his latest work on xLSTM, explaining how exponential gating and matrix memory replace the old sigmoid bottleneck to create architectures that can compete with Transformers on sequence modeling while maintaining fixed memory footprints.

The conversation covers the practical advantages of xLSTM for industrial applications like robotics simulation and discrete element modeling, where constant memory consumption matters more than raw benchmark performance. Hochreiter also discusses the neuro-symbolic Pi AI project at JKU Linz, his view that genuine reasoning requires something beyond pattern matching on training data, and why he believes the current scaling paradigm will hit a wall. The technical deep-dive into FlashAttention comparisons, recurrent vs attention-based architectures, and the evolution from sigmoid to exponential gating mechanisms gives a clear picture of where LSTM-style architectures fit in the current landscape.

---
REFERENCES:
Paper:
[00:00:13] Long Short-Term Memory (Original LSTM Paper)
https://direct.mit.edu/neco/article-abstract/9/8/1735/6109/Long-Short-Term-Memory
[00:04:18] Kolmogorov Complexity
link.springer.com/article/10.1007/BF02478259
[00:19:38] xLSTM: Extended Long Short-Term Memory
arxiv.org/abs/2405.04517
[00:22:53] FlashAttention: Fast and Memory-Efficient Attention
arxiv.org/abs/2205.14135
[00:36:08] Mamba: Linear-Time Sequence Modeling
arxiv.org/abs/2312.00752
[00:55:03] Core Knowledge Theory of Human Cognition
harvardlds.org/wp-content/uploads/2017/01/SpelkeKinzler07-1.pdf
Book:
[00:53:18] Thinking, Fast and Slow
amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555
Company:
[01:00:18] NXAI: Industrial AI with xLSTM
https://investinaustria.at/en/blog/nxai-top-researchers-in-austria-develop-solutions-for-industrial-companies/

---
LINKS:
Full Transcript: app.rescript.info/share/50e3f79d116b1a92aa40819b6cb40eb2
Download PDF transcript: app.rescript.info/api/public/sessions/bdb8e291a0e4514d/pdf

Prof. Sepp Hochreiter
nx-ai.com
https://x.com/hochreitersepp
https://scholar.google.at/citations?user=tvUH3WMAAAAJ&hl=en
LSTM: The Comeback Story? [Prof. Sepp Hochreiter]New Research: LMArena is riggedAutoGrad Changed Everything (Not Transformers) [Dr. Jeff Beck]Intelligence is collective, not artificial — Prof. Michael I. Jordan (UC Berkeley / Inria)The Final Boss of Deep LearningWere Already Living Inside AI machines (And Dont Know It) - Christopher SummerfieldAI in 2024 - efficiency over model size (Nick Jakobi)LLMs Cosplay Understanding (Jeremy Howard)The Dream Chocolate That Stopped Being Real (Joscha Bach)Test-Time Adaptation: A New Frontier in AI [Dr. Jonas Hübotter]The scientist who coined retrieval augmented generationThe Gap Between Humans and Machines Is   [Dr. Max Bartolo]
Machine Learning Street Talk |

LSTM: The Comeback Story? [Prof. Sepp Hochreiter]

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER