Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
Authors: Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, Viveck Cadambe
NeurIPS 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Finally, we validate the theory with experimental results, running over AWS EC2 clouds and an internal GPU cluster. |
| Researcher Affiliation | Academia | Farzin Haddadpour Penn State fxh18@psu.edu Mohammad Mahdi Kamani Penn State mqk5591@psu.edu Mehrdad Mahdavi Penn State mzm616@psu.edu Viveck R. Cadambe Penn State vxc12@psu.edu |
| Pseudocode | Yes | Algorithm 1 LUPA-SGD(τ): Local updates with periodic averaging. |
| Open Source Code | Yes | The implementation code is available at https://github.com/mmkamani7/LUPA-SGD. |
| Open Datasets | Yes | Epsilon dataset2, a popular large scale binary dataset, consists of 400, 000 training samples and 100, 000 test samples with feature dimension of 2000. (Footnote 2: https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html) |
| Dataset Splits | No | The paper mentions 400,000 training samples and 100,000 test samples but does not specify a validation split or details on how data was partitioned for validation. |
| Hardware Specification | Yes | Most of the experiments will be run on Amazon EC2 cluster with 5 p2.xlarge instances. ... We also use an internal high performance computing (HPC) cluster equipped with NVIDIA Tesla V100 GPUs. |
| Software Dependencies | No | The paper mentions software like PyTorch, TensorFlow, mpi4py, and Open MPI, but it does not specify version numbers for any of these dependencies, which is required for reproducibility. |
| Experiment Setup | Yes | The learning rate and regularization parameter are 0.01 and 1 10 4, respectively, and the minibatch size is 128 unless otherwise stated. |