Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
The Non-IID Data Quagmire of Decentralized Machine Learning
Authors: Kevin Hsieh, Amar Phanishayee, Onur Mutlu, Phillip Gibbons
ICML 2020 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this paper, we take a step toward better understanding this challenge by presenting a detailed experimental study of decentralized DNN training on a common type of data skew: skewed distribution of data labels across devices/locations. |
| Researcher Affiliation | Collaboration | 1Microsoft Research 2Carnegie Mellon University 3ETH Zürich. |
| Pseudocode | Yes | Gaia (Hsieh et al., 2017)... (Algorithm 1 in Appendix A1). Federated Averaging (Mc Mahan et al., 2017)... (Algorithm 2 in Appendix A). Deep Gradient Compression (Lin et al., 2018)... (Algorithm 3 in Appendix A). |
| Open Source Code | Yes | All source code and settings are available at https: //github.com/kevinhsieh/non_iid_dml. |
| Open Datasets | Yes | We use two datasets, CIFAR10 (Krizhevsky, 2009) and Image Net (Russakovsky et al., 2015)... To facilitate further study on skewed label partitions, we release a real-world, geo-tagged dataset of common mammals on Flickr (Flickr), which is openly available at https://doi.org/10.5281/ zenodo.3676081 ( 2.2). |
| Dataset Splits | Yes | We use the default validation set of each of the two datasets to quantify the validation accuracy as our model quality metric... We control the skewness by controlling the fraction of data that are non-IID. For example, 20% non-IID indicates 20% of the dataset is partitioned by labels, while the remaining 80% is partitioned uniformly at random. |
| Hardware Specification | No | The paper mentions running experiments on a "GPU parameter server system" but does not provide specific hardware details such as GPU models, CPU types, or memory specifications. |
| Software Dependencies | No | The paper mentions using "Caffe" but does not specify a version number or list other software dependencies with version information. |
| Experiment Setup | Yes | For all applications, we tune the training parameters (e.g., learning rate, minibatch size, number of epochs, etc.)... Appendix C lists all major training parameters in our study. |