Practical Hybrid Gradient Compression for Federated Learning Systems
Authors: Sixu Hu, Linshan Jiang, Bingsheng He
IJCAI 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The extensive theoretical and empirical analysis demonstrates the effectiveness of our framework in achieving a high compression ratio for both uplink and downlink communications with negligible loss of model accuracy, surpassing the state-of-the-art compression methods. Section 5 presents the empirical analysis. We use Fed Avg [Mc Mahan et al., 2017] as the baseline algorithm and evaluate our method using Res Net-18 [He et al., 2016] on CIFAR-10 [Krizhevsky et al., 2009] and two-layer LSTM [Press and Wolf, 2017] on Penn-Tree Bank (PTB) [Marcus et al., 1993]. |
| Researcher Affiliation | Academia | Sixu Hu , Linshan Jiang , Bingsheng He National University of Singapore sixuhu@comp.nus.edu.sg, linshan@nus.edu.sg, hebs@comp.nus.edu.sg |
| Pseudocode | Yes | A more detailed pseudo code implementation is provided in Appendix A.3. |
| Open Source Code | No | The paper does not provide an explicit statement about releasing source code for the described methodology or a link to a code repository. |
| Open Datasets | Yes | We use Fed Avg [Mc Mahan et al., 2017] as the baseline algorithm and evaluate our method using Res Net-18 [He et al., 2016] on CIFAR-10 [Krizhevsky et al., 2009] and two-layer LSTM [Press and Wolf, 2017] on Penn-Tree Bank (PTB) [Marcus et al., 1993]. The experiments in Figure 4 and 5 are performed with a 4-layer CNN on the FEMNIST [Caldas et al., 2018] dataset. |
| Dataset Splits | No | The paper mentions using CIFAR-10, Penn-Tree Bank, and FEMNIST datasets and describes how clients are sampled for FEMNIST, but it does not explicitly provide specific training, validation, or test dataset splits (e.g., percentages, sample counts, or explicit confirmation of standard splits). |
| Hardware Specification | Yes | We conduct experiments on a server with two Intel Xeon Gold 6248 CPUs @ 2.50GHz, 192GB of memory @ 3200MHz, and four Nvidia Tesla V100 GPUs. |
| Software Dependencies | No | The paper mentions PyTorch in an appendix figure caption ('All models are trained with PyTorch') but does not specify its version number or any other software dependencies with their specific versions. |
| Experiment Setup | Yes | Unless otherwise specified, we deploy the FL system with four clients with a learning rate of 0.1 for CIFAR-10, and a learning rate of 20 for PTB. We reduce the learning rate by a factor of 0.1 at 50% and 75% of the training process. For algorithms involving sparsification, we use the same sparsification ratio warm-up technique to speed up the training process. Specifically, we exponentially step down the sparsification ratio from 0.25 to the target in 300 iterations. |