OpenAI Astra and Recurrent Depth / Looped Transformers @SebastianRaschka
OpenAI Astra and Recurrent Depth / Looped Transformers  @SebastianRaschka
Uploaded September 2026 | Updated November 2026, 1 day ago
A short video explaining the rumored recurrent depth (looped transformer) architectural design in OpenAI's new (upcoming) Astra model.

In particular, I discuss what looped transformers are and go over the related literature.

UPDATE: Expanding on the video, I also put together a longer write-up on the topic: magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and

Other links and references

The Information article on OpenAI Astra:
theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns

Looped depth sharing explainer:
sebastianraschka.com/llm-architecture-gallery/looped-depth-sharing

LLM Architecture Gallery:
sebastianraschka.com/llm-architecture-gallery

Nanbeige4.2-3B technical report:
arxiv.org/abs/2607.22083

Mixture-of-Recursions:
arxiv.org/abs/2507.10524

Universal Transformers:
arxiv.org/abs/1807.03819

Scaling up Test-Time Compute with Latent Reasoning:
arxiv.org/abs/2502.05171

00:00 OpenAI Astra and the Recurrent-Depth Rumor
01:37 Claims About Recurrent Depth
03:02 Nanbeige4.2-3B’s Looped Transformer
05:21 Why Reuse the Same Layers?
07:43 Training From Scratch vs. Retrofitting
09:47 Why Use Two Passes?
10:35 KV-Cache Implications
12:00 Mixture-of-Recursions and Dynamic Depth
16:18 Validation-Loss Comparison
18:16 Attention and KV Caching With Dynamic Depth
19:55 Recurrent Depth and Chain of Thought
23:26 Recursive Self-Improvement
24:27 Universal Transformers
25:27 Latent Reasoning With Recurrent Depth
27:09 Summary

Corrections:
3:09 I incorrectly said that Nanbeige was developed in India. Nanbeige4.2-3B was developed by Nanbeige LLM Lab and Boss Zhipin in China. I had confused it with Sarvam AI, which is based in India.

If you enjoyed this video, I also write about LLM research and model architectures in my Ahead of AI newsletter:
magazine.sebastianraschka.com
OpenAI Astra and Recurrent Depth / Looped Transformers13.4.5 Sequential Feature Selection   Code Examples (L13: Feature Selection)L15.7 An RNN Sentiment Classifier in PyTorchBuild A Reasoning Model From Scratch 1: Motivation & Code SetupL8.6 Multinomial Logistic Regression / Softmax RegressionWhere verifiable rewards for reasoning model training come fromHow top-p filters tokensL5.6 Understanding Gradient DescentL19.5.2.2 GPT-v1: Generative Pre-Trained TransformerL6.1 Learning More About PyTorchL14.4.1 Replacing Max-Pooling with Convolutional LayersDeep Learning News #9, Mar 27 2021
Sebastian Raschka |

OpenAI Astra and Recurrent Depth / Looped Transformers

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER