Uploaded June 2021 | Updated September 2026, 12 hours ago
Training a large model with Machine Learning used to be strictly limited by GPU memory, but now with Microsoft's new paper, ZeRO-Infinity, training much larger ML models is possible. ZeRO-Infinity allows training models with trillions of parameters with a modest amount of GPUs and finetuning many billion parameter models with just one GPU. This is a great advancement for anyone who wants to work with big models like GPT-2 and the like! Use ML train large model.
ZeRO-Infinity blog post: microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training
ZeRO-Infinity paper: arxiv.org/abs/2104.07857
DeepSpeed: deepspeed.ai
Training a large model with Machine Learning used to be strictly limited by GPU memory, but now with Microsoft's new paper, ZeRO-Infinity, training much larger ML models is possible. ZeRO-Infinity allows training models with trillions of parameters with a modest amount of GPUs and finetuning many billion parameter models with just one GPU. This is a great advancement for anyone who wants to work with big models like GPT-2 and the like! Use ML train large model.
ZeRO-Infinity blog post: microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training
ZeRO-Infinity paper: arxiv.org/abs/2104.07857
DeepSpeed: deepspeed.ai
