FetchSGD: Communication-Efficient Federated Learning with Sketching
Authors: Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, Raman Arora
ICML 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We prove that Fetch SGD has favorable convergence guarantees, and we demonstrate its empirical effectiveness by training two residual networks and a transformer model. |
| Researcher Affiliation | Collaboration | 1University of California, Berkeley, California, USA 2Johns Hopkins University, Baltimore, Maryland 3Amazon. |
| Pseudocode | Yes | Fetch SGD is presented in full in Algorithm 1. |
| Open Source Code | Yes | Code available at https://github.com/ kiddyboots216/Comm Efficient. Git commit at the time of camera-ready: 833ca44. |
| Open Datasets | Yes | CIFAR10 and CIFAR100 (Krizhevsky et al., 2009)... Federated EMNIST... (Caldas et al., 2018)... We finetune a pretrained GPT2 on the Persona Chat dataset... (Zhang et al., 2018). |
| Dataset Splits | Yes | CIFAR10 (CIFAR100) we use 10,000 (50,000) clients... We report accuracy on the test datasets... We report final accuracies on the validation dataset. |
| Hardware Specification | No | The paper does not specify the exact hardware (e.g., CPU, GPU models, cloud instance types) used for running the experiments. |
| Software Dependencies | No | We implement and compare Fetch SGD, gradient sparsification (local top-k), and Fed Avg using Py Torch (Paszke et al., 2019). |
| Experiment Setup | Yes | Details on which hyperparameters were chosen for each task can be found in Appendix A.1. |