Layer-Wise Adaptive Model Aggregation for Scalable Federated Learning
Authors: Sunwoo Lee, Tuo Zhang, A. Salman Avestimehr
AAAI 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our extensive empirical study shows that, as the aggregation interval increases, Fed LAMA shows a remarkably smaller accuracy drop than the periodic full aggregation, while achieving comparable communication efficiency. Experiments Experimental Settings We evaluate Fed LAMA using three representative benchmarks: CIFAR-10 (Res Net20 (He et al. 2016)), CIFAR-100 (Wide Res Net28-10 (Zagoruyko and Komodakis 2016)), and Federated Extended MNIST (CNN (Caldas et al. 2018)). |
| Researcher Affiliation | Academia | Sunwoo Lee1,2, Tuo Zhang1, and Salman Avestimehr1 1University of Southern California, USA 2Inha University, South Korea {sunwool,tuozhang,avestime}@usc.edu |
| Pseudocode | Yes | Algorithm 1: Layer-wise Aggregation Interval Adjustment. Algorithm 2: Fed LAMA: Federated Layer-wise Adaptive Model Aggregation. |
| Open Source Code | No | The paper does not provide any specific link or statement regarding the availability of its own source code for the methodology described. |
| Open Datasets | Yes | We evaluate Fed LAMA using three representative benchmarks: CIFAR-10 (Krizhevsky 2009), CIFAR-100, and Federated Extended MNIST (Cohen et al. 2017). |
| Dataset Splits | Yes | We split CIFAR datasets to 128 subsets using Dirichlet distributions. The concentration coefficient is 0.1 which represents a high-degree of non-IIDness. The distribution is generated with respect to the labels, and the local dataset sizes are all different (unbalanced non-IID). As for FEMNIST, we randomly select 256 writers samples (~ 7.3% of the whole samples) and assign 2 writers samples to each client. The total number of local steps is 10,000 and the local batch size is 32. |
| Hardware Specification | Yes | All our experiments are conducted on 8 NVIDIA A1000 GPUs. |
| Software Dependencies | Yes | We use Tensor Flow 2.4.3 for local training and MPI for model aggregation. |
| Experiment Setup | Yes | Experimental Settings We evaluate Fed LAMA using three representative benchmarks: CIFAR-10 (Res Net20 (He et al. 2016)), CIFAR-100 (Wide Res Net28-10 (Zagoruyko and Komodakis 2016)), and Federated Extended MNIST (CNN (Caldas et al. 2018)). We use Tensor Flow 2.4.3 for local training and MPI for model aggregation. The total number of local steps is 10,000 and the local batch size is 32. Non-IID Datasets We split CIFAR datasets to 128 subsets using Dirichlet distributions. The concentration coefficient is 0.1 which represents a high-degree of non-IIDness. In all the experiments, random 25% of 128 clients run each communication round. (Also refers to LR, τ, ϕ in tables 1, 2, 3, 4) |