Uploaded August 2025 | Updated September 2026, 1 week ago

---

Llama 4's pre-training scale is growing exponentially, with 100K GPUs used, a 6x increase from its predecessor. Initializing training takes longer, and failure probability increases with larger scale. Training throughput aka Effective Training time degrades significantly as a result.

To address these challenges, researchers are experimenting in parallel for faster initialization of large scale jobs, and fault-tolerant paradigms.

Learn more here: atscaleconference.com