FLuID: Mitigating Stragglers in Federated Learning using Invariant Dropout
Authors: Irene Wang, Prashant Nair, Divya Mahajan
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate FLu ID using five real-world mobile clients. The evaluations show that Invariant Dropout maintains baseline model efficiency while alleviating the performance bottleneck of stragglers through a dynamic and lightweight runtime approach. |
| Researcher Affiliation | Collaboration | Irene Wang1,3, Prashant J. Nair1, Divya Mahajan2,3 University of British Columbia1, Microsoft2, Georgia Institute of Technology3 |
| Pseudocode | Yes | Algorithm 1 outlines the FLu ID framework. |
| Open Source Code | Yes | Source code is available at https://github.com/iwang05/FLu ID |
| Open Datasets | Yes | The FEMNIST datasets consist of images of numbers and letters, partitioned based on the writer of the character in non-IID setting. [...] The Shakespeare dataset partitions data based on roles in Shakespeares plays in non-IID setting. [...] The CIFAR10 dataset consists of images, partitioned using the same strategy as Fj ORD [HLA+21] and the IID partition provided by the Flower [BTM+20]. |
| Dataset Splits | No | The paper mentions partitioning strategies for datasets (non-IID, IID) and refers to prior work for partitioning. However, it does not provide specific train/validation/test split percentages, sample counts, or explicit details about how the validation set was created or used for model selection/hyperparameter tuning, beyond general mentions of accuracy evaluation during training. |
| Hardware Specification | Yes | Table 1: Software-Hardware specifications of clients Device Year Android Version CPU (Cores) LG Velvet 5G 2020 10 1 2.4 GHz Kryo 475 Prime + 1 2.2 GHz Kryo 475 Gold + 6 1.8 GHz Kryo 475 Silver Google Pixel 3 2018 9 4 2.5 GHz Kryo 385 Gold + 4 1.6 GHz Kryo 385 Silver Samsung Galaxy S9 2018 10 4 2.8 GHz Kryo 385 Gold + 4 1.7 GHz Kryo 385 Silver Samsung Galaxy S10 2019 11 2 2.73 GHz Mongoose M4 + 2 2.31 GHz Cortex-A75 + 4 1.95 GHz Cortex-A55 Google Pixel 4 2019 12 1 2.84 GHz Kryo 485 + 3 2.42 GHz Kryo 485 + 4 1.78 GHz Kryo 485 |
| Software Dependencies | Yes | FLu ID is implemented on top of the Flower (v0.18.0) [BTM+20] framework and Tensor Flow Lite [Goo] from Tensor Flow v2.8.0 [ABC+16]. |
| Experiment Setup | Yes | We trained the models for 100, 250, and 65 epochs for CIFAR10, FEMNIST, and Shakespeare datasets, respectively. [...] For this example, we choose thresholds of 180%, 10%, and 500%, respectively, for these three datasets and compute their invariant neurons. [...] Table 3 presents the percentage of invariant neurons observed at different threshold values, and the overall training accuracy of the FEMNIST model, using a sub-model size of 0.75 for the stragglers. [...] Table 4 presents the accuracy when stragglers are assigned to 4 equal-sized clusters (sub-model size 0.65, 0.75, 0.85, 0.95). [...] The accuracy results of varying the straggler ratios while using 0.75-sized sub-models are summarized in Figure 8. [...] We scaled FLu ID to 1,000 clients with the FEMNIST dataset for 500 global training rounds. We run with a client sampling ratio of 10%... |