Communication-Efficient Topologies for Decentralized Learning with $O(1)$ Consensus Rate
Authors: Zhuoqing Song, Weijian Li, Kexin Jin, Lei Shi, Ming Yan, Wotao Yin, Kun Yuan
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This section presents experimental results to validate Equi Topo s network-size-independent consensus rate and its comparison with other commonly-used topologies in DSGD on both strongly-convex problems and non-convex deep learning tasks. |
| Researcher Affiliation | Collaboration | 1Fudan University 2Alibaba DAMO Academy 3Princeton University 4The Chinese University of Hong Kong, Shenzhen 5Michigan State University 6Peking University 7Shanghai Artificial Intelligence Laboratory |
| Pseudocode | Yes | Algorithm 1: OD-Equi Dyn weight matrix generation at iteration t |
| Open Source Code | Yes | Our code is implemented through Blue Fog and available at https://github.com/kexinjinnn/Equi Topo. |
| Open Datasets | Yes | We consider the image classification task with Res Net20 model [9] over the CIFAR-10 dataset [14]. ... Experiments with Equi Dyn topologies and results on MNIST dataset [15] are in Appendix D. |
| Dataset Splits | No | The details are in the Appendix. (The main text does not specify explicit dataset splits, only mentions that training details are in the Appendix.) |
| Hardware Specification | Yes | We utilize Blue Fog [38] to support decentralized communication and topology setting in a cluster of 17 Tesla P100 GPUs. |
| Software Dependencies | No | We utilize Blue Fog [38] to support decentralized communication and topology setting in a cluster of 17 Tesla P100 GPUs. (The paper mentions Blue Fog, but does not provide specific version numbers for any software dependencies.) |
| Experiment Setup | No | Fig. 5 depicts that D/U-Equi Static converges much faster than a static exponential graph, especially in the initial stages when the learning rate is large. (The paper generally refers to training details, including hyperparameters, being in Appendix D, but does not specify them in the main text.) |