Uploaded September 2026 | Updated November 2026, 1 day ago
A short video explaining the rumored recurrent depth (looped transformer) architectural design in OpenAI's new (upcoming) Astra model.
In particular, I discuss what looped transformers are and go over the related literature.
UPDATE: Expanding on the video, I also put together a longer write-up on the topic: magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and
Other links and references
The Information article on OpenAI Astra:
theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns
Looped depth sharing explainer:
sebastianraschka.com/llm-architecture-gallery/looped-depth-sharing
LLM Architecture Gallery:
sebastianraschka.com/llm-architecture-gallery
Nanbeige4.2-3B technical report:
arxiv.org/abs/2607.22083
Mixture-of-Recursions:
arxiv.org/abs/2507.10524
Universal Transformers:
arxiv.org/abs/1807.03819
Scaling up Test-Time Compute with Latent Reasoning:
arxiv.org/abs/2502.05171
00:00 OpenAI Astra and the Recurrent-Depth Rumor
01:37 Claims About Recurrent Depth
03:02 Nanbeige4.2-3B’s Looped Transformer
05:21 Why Reuse the Same Layers?
07:43 Training From Scratch vs. Retrofitting
09:47 Why Use Two Passes?
10:35 KV-Cache Implications
12:00 Mixture-of-Recursions and Dynamic Depth
16:18 Validation-Loss Comparison
18:16 Attention and KV Caching With Dynamic Depth
19:55 Recurrent Depth and Chain of Thought
23:26 Recursive Self-Improvement
24:27 Universal Transformers
25:27 Latent Reasoning With Recurrent Depth
27:09 Summary
Corrections:
3:09 I incorrectly said that Nanbeige was developed in India. Nanbeige4.2-3B was developed by Nanbeige LLM Lab and Boss Zhipin in China. I had confused it with Sarvam AI, which is based in India.
If you enjoyed this video, I also write about LLM research and model architectures in my Ahead of AI newsletter:
magazine.sebastianraschka.com
A short video explaining the rumored recurrent depth (looped transformer) architectural design in OpenAI's new (upcoming) Astra model.
In particular, I discuss what looped transformers are and go over the related literature.
UPDATE: Expanding on the video, I also put together a longer write-up on the topic: magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and
Other links and references
The Information article on OpenAI Astra:
theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns
Looped depth sharing explainer:
sebastianraschka.com/llm-architecture-gallery/looped-depth-sharing
LLM Architecture Gallery:
sebastianraschka.com/llm-architecture-gallery
Nanbeige4.2-3B technical report:
arxiv.org/abs/2607.22083
Mixture-of-Recursions:
arxiv.org/abs/2507.10524
Universal Transformers:
arxiv.org/abs/1807.03819
Scaling up Test-Time Compute with Latent Reasoning:
arxiv.org/abs/2502.05171
00:00 OpenAI Astra and the Recurrent-Depth Rumor
01:37 Claims About Recurrent Depth
03:02 Nanbeige4.2-3B’s Looped Transformer
05:21 Why Reuse the Same Layers?
07:43 Training From Scratch vs. Retrofitting
09:47 Why Use Two Passes?
10:35 KV-Cache Implications
12:00 Mixture-of-Recursions and Dynamic Depth
16:18 Validation-Loss Comparison
18:16 Attention and KV Caching With Dynamic Depth
19:55 Recurrent Depth and Chain of Thought
23:26 Recursive Self-Improvement
24:27 Universal Transformers
25:27 Latent Reasoning With Recurrent Depth
27:09 Summary
Corrections:
3:09 I incorrectly said that Nanbeige was developed in India. Nanbeige4.2-3B was developed by Nanbeige LLM Lab and Boss Zhipin in China. I had confused it with Sarvam AI, which is based in India.
If you enjoyed this video, I also write about LLM research and model architectures in my Ahead of AI newsletter:
magazine.sebastianraschka.com










