EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning
Authors: Shay Vargaftik, Ran Ben Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben Itzhak, Michael Mitzenmacher
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We derive appealing theoretical guarantees for EDEN and evaluate it empirically. Our results demonstrate that EDEN consistently improves over state-of-the-art DME techniques. We implement and evaluate EDEN in Py Torch (Paszke et al., 2019) and Tensor Flow (Abadi et al., 2015)2 and show that EDEN can compress vectors with more than 67 million coordinates within 61 ms. Compared with state-of-the-art DME techniques, EDEN consistently provides better mean estimation, which translates to higher accuracy in various federated and distributed learning tasks and scenarios. |
| Researcher Affiliation | Collaboration | 1VMware Research 2University College London 3Ben-Gurion University 4Stanford University 5Harvard University. |
| Pseudocode | No | The paper describes procedures and mathematical formulations but does not include any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our Py Torch and Tensor Flow implementations are available as open source at https://github.com/amitport/EDENDistributed-Mean-Estimation. |
| Open Datasets | Yes | We evaluate EDEN over the federated versions of the EMNIST (Cohen et al., 2017) image classification task and the Shakespeare (Shakespeare) next-word prediction task. Specifically, federated EMNIST (Caldas et al., 2018a) includes 749,068 handwritten characters partitioned among their 3400 writers (i.e., this is the total number of clients), and federated Shakespeare (Mc Mahan et al., 2017) consists of 18,424 lines of text from Shakespeare plays partitioned among the respective 715 speakers (i.e., clients). |
| Dataset Splits | No | The paper describes client partitioning for federated learning tasks, but it does not explicitly state the specific train/validation/test dataset splits (e.g., percentages or counts) or the methodology used for such splits. |
| Hardware Specification | Yes | Our encoding speed measurements are performed using NVIDIA Ge Force RTX 3090 GPU. The machine has Intel Core i9-10980XE CPU (18 cores, 3.00 GHz, and 24.75 MB cache) and 128 GB RAM. |
| Software Dependencies | Yes | Our Py Torch and Tensor Flow implementations are available as open source at https://github.com/amitport/EDENDistributed-Mean-Estimation. We use Ubuntu 20.04.2 LTS operating system, CUDA release 11.1 (V11.1.105), and Py Torch version 1.10.1. |
| Experiment Setup | Yes | We run Fed Avg (Mc Mahan et al., 2017) with the Adam server optimizer (Kingma & Ba, 2015) and sample n = 10 clients per round. We re-use code, client partitioning, models, and hyperparameters from the federated learning benchmark of Reddi et al. (2021). Those are restated for convenience in Appendix I.2. Table 1. Hyperparameters for the EMNIST and Shakespeare experiments. Task Clients per round Rounds Batch size Client lr Server lr Adam s ϵ EMNIST 10 1500 20 10 1.5 10 2.5 10 4 Shakespeare 10 1200 4 1 10 2 10 3 |