Communication-efficient Distributed SGD with Sketching
Authors: Nikita Ivkin, Daniel Rothchild, Enayat Ullah, Vladimir braverman, Ion Stoica, Raman Arora
NeurIPS 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We run experiments on a transformer model, an LSTM, and a residual network, demonstrating up to a 40x reduction in total communication cost with no loss in final model performance. We also show experimentally that SKETCHED-SGD scales to at least 256 workers without increasing communication cost or degrading model performance. |
| Researcher Affiliation | Collaboration | Nikita Ivkin Amazon ivkin@amazon.com Daniel Rothchild UC Berkeley drothchild@berkeley.edu Enayat Ullah Johns Hopkins University enayat@jhu.edu Vladimir Braverman Johns Hopkins University vova@cs.jhu.edu Ion Stoica UC Berkeley istoica@berkeley.edu Raman Arora Johns Hopkins University arora@cs.jhu.edu |
| Pseudocode | Yes | Algorithm 1 HEAVYMIX, Algorithm 2 SKETCHED-SGD, Algorithm 3 EMPIRICAL TRAINING |
| Open Source Code | Yes | 4Code is available at https://github.com/dhroth/sketchedsgd |
| Open Datasets | Yes | We train both models on the WMT 2014 English to German translation task...We train a residual network on the CIFAR-10 dataset |
| Dataset Splits | No | The paper mentions using CIFAR-10 and WMT 2014 datasets and references 'Validation Perplexity', but does not provide specific details on how the validation dataset split was created (e.g., percentages or sample counts carved out from the training set). |
| Hardware Specification | No | The paper mentions running experiments 'on the GPU' but does not specify any particular GPU model (e.g., NVIDIA A100, Tesla V100) or other hardware specifications such as CPU model or memory. |
| Software Dependencies | No | We implement a parallelized Count Sketch with Py Torch [Paszke et al., 2017]. |
| Experiment Setup | Yes | For all SKETCHED-SGD experiments on these two models, we use k = 100, 000...5Sketch size: 5 rows by 1M columns; P = 36. 6Sketch size: 15 rows by 180,000 columns; P = 16. 7Sketch size: 5 rows by 180,000 columns, P = 26 |