CO2: Efficient Distributed Training with Full Communication-Computation Overlap
CO2: Efficient Distributed Training with Full Communication-Computation Overlap
About this item
Full title
Author / Creator
Sun, Weigao , Qin, Zhen , Sun, Weixuan , Li, Shidi , Li, Dong , Shen, Xuyang , Yu, Qiao and Zhong, Yiran
Publisher
Ithaca: Cornell University Library, arXiv.org
Journal title
Language
English
Formats
Publication information
Publisher
Ithaca: Cornell University Library, arXiv.org
Subjects
More information
Scope and Contents
Contents
The fundamental success of large language models hinges upon the efficacious implementation of large-scale distributed training techniques. Nevertheless, building a vast, high-performance cluster featuring high-speed communication interconnectivity is prohibitively costly, and accessible only to prominent entities. In this work, we aim to lower thi...
Alternative Titles
Full title
CO2: Efficient Distributed Training with Full Communication-Computation Overlap
Authors, Artists and Contributors
Author / Creator
Identifiers
Primary Identifiers
Record Identifier
TN_cdi_proquest_journals_2919919576
Permalink
https://devfeature-collection.sl.nsw.gov.au/record/TN_cdi_proquest_journals_2919919576
Other Identifiers
E-ISSN
2331-8422
