Uploaded June 2026 | Updated September 2026, 3 weeks ago
Di-PS: System-Algorithm Co-Design for Asynchronous and Heterogeneous Cross-cluster LLM Training at Scale
Shengwei Li, National Key Laboratory of Parallel and Distributed Computing; Qiaoling Chen, Shanghai Artificial Intelligence Laboratory and Nanyang Technology University; Zhiquan Lai, National Key Laboratory of Parallel and Distributed Computing; Penglong Jiao, Wenwen Qu, Kun Cai, Jiaxing Li, Peng Sun, and Xingcheng Zhang, Shanghai Artificial Intelligence Laboratory; Xiaoge Deng, Dongsheng Li, and Kai Lu, National Key Laboratory of Parallel and Distributed Computing; Tianwei Zhang, Nanyang Technological University
Large language models (LLMs) have revolutionized artificial intelligence, exhibiting remarkable performance in various tasks. Training these models demands extensive computational resources, which are often economically and physically prohibitive. Cross-cluster training can balance infrastructure costs, alleviate physical and resource constraints, better match workload demands, and sustain higher efficiency through geo-distributed deployment. However, challenges arise from network variability, heterogeneous computational resources, and intrinsic training instability.
To address these issues, we present Di-PS, a novel framework for cross-cluster LLM training at scale. The core of Di-PS is the system-algorithm co-design of a parameter server paradigm, to achieve heterogeneous, asynchronous, and resilient training across decentralized clusters. We make several innovative contributions in Di-PS, including (i) an efficient parameter server design for cross-cluster communication of LLM parameters, (ii) a pseudo-gradient penalty strategy for convergence stability enhancement of asynchronous two-stage optimization, and (iii) a resilience mechanism for fault tolerance in cross-cluster training. Results from the controlled experimental setting demonstrate that Di-PS improves training efficiency by up to 4.67× over synchronous cross-cluster approaches while maintaining model quality, and achieving near-linear scalability in heterogeneous training resources. Di-PS has been deployed in the production environment, involving dynamic training scales with up to 9 clusters and more than 10,000 NPUs. At this scale, Di-PS enables successful cross-cluster training of a 100B-parameter LLM with only 6% overhead compared to single-cluster training, and effectively handles frequent failures and resource changes.
View the full NSDI '26 program at usenix.org/conference/nsdi26/technical-sessions
Di-PS: System-Algorithm Co-Design for Asynchronous and Heterogeneous Cross-cluster LLM Training at Scale
Shengwei Li, National Key Laboratory of Parallel and Distributed Computing; Qiaoling Chen, Shanghai Artificial Intelligence Laboratory and Nanyang Technology University; Zhiquan Lai, National Key Laboratory of Parallel and Distributed Computing; Penglong Jiao, Wenwen Qu, Kun Cai, Jiaxing Li, Peng Sun, and Xingcheng Zhang, Shanghai Artificial Intelligence Laboratory; Xiaoge Deng, Dongsheng Li, and Kai Lu, National Key Laboratory of Parallel and Distributed Computing; Tianwei Zhang, Nanyang Technological University
Large language models (LLMs) have revolutionized artificial intelligence, exhibiting remarkable performance in various tasks. Training these models demands extensive computational resources, which are often economically and physically prohibitive. Cross-cluster training can balance infrastructure costs, alleviate physical and resource constraints, better match workload demands, and sustain higher efficiency through geo-distributed deployment. However, challenges arise from network variability, heterogeneous computational resources, and intrinsic training instability.
To address these issues, we present Di-PS, a novel framework for cross-cluster LLM training at scale. The core of Di-PS is the system-algorithm co-design of a parameter server paradigm, to achieve heterogeneous, asynchronous, and resilient training across decentralized clusters. We make several innovative contributions in Di-PS, including (i) an efficient parameter server design for cross-cluster communication of LLM parameters, (ii) a pseudo-gradient penalty strategy for convergence stability enhancement of asynchronous two-stage optimization, and (iii) a resilience mechanism for fault tolerance in cross-cluster training. Results from the controlled experimental setting demonstrate that Di-PS improves training efficiency by up to 4.67× over synchronous cross-cluster approaches while maintaining model quality, and achieving near-linear scalability in heterogeneous training resources. Di-PS has been deployed in the production environment, involving dynamic training scales with up to 9 clusters and more than 10,000 NPUs. At this scale, Di-PS enables successful cross-cluster training of a 100B-parameter LLM with only 6% overhead compared to single-cluster training, and effectively handles frequent failures and resource changes.
View the full NSDI '26 program at usenix.org/conference/nsdi26/technical-sessions
