Communication-Efficient Federated Hypergradient Computation via Aggregated Iterative Differentiation
Authors: Peiyao Xiao, Kaiyi Ji
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our extensive experiments demonstrate the great effectiveness and communication efficiency of the proposed method. The right plot compares the performance among fully local hypergradient estimator (i.e., using only local information), AID-based FHE and Agg ITD in federeated hyper-representation learning in the presence of data heterogeneity. In this section, we compare the performance of the proposed FBO-Agg ITD method with Fed Nest and LFed Nest in Tarzanagh et al. 2022 on a hyper-representation problem. |
| Researcher Affiliation | Academia | 1Department of Computer Science and Engineering, University at Buffalo, Baffalo, USA. Correspondence to: Kaiyi Ji <kaiyiji@buffalo.edu>. |
| Pseudocode | Yes | Algorithm 1 Agg ITD(x, y, β) Algorithm 2 y+ = One-Round-Lower(x, y, q, β) Algorithm 3 FBO-Agg ITD Algorithm 4 x+ = One-Round-Upper(x, y+, h, α) |
| Open Source Code | No | The paper states: "Since no codes are provided in (Yang et al., 2022), we wrote a code for comparison." This indicates they wrote code for a comparison, but they do not explicitly state that the code for *their* proposed method is open-source or provide a link for it. |
| Open Datasets | Yes | We study the impact of data heterogeneity on the comparison algorithms by considering both the i.i.d. and non-i.i.d. ways of data partitioning of MNIST, following the setup in Mc Mahan et al. 2017. Finally, Figure 5 shows the performance of FBO-Agg ITD on CIFAR-10 with MLP/CNN network in the i.i.d. setting. |
| Dataset Splits | No | The paper mentions using MNIST and CIFAR-10 and discusses i.i.d. and non-i.i.d. data partitioning, but it does not specify explicit training, validation, or test split percentages or counts. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for experiments, such as CPU or GPU models, or memory specifications. |
| Software Dependencies | No | The paper mentions "torch.autograd" but does not specify its version or any other software dependencies with version numbers. It mentions "SVRG-type optimizer" and "SGD-type optimizer" as types, but not specific software libraries or their versions. |
| Experiment Setup | Yes | Parameter selection for the experiments in Figure 2 and Figure 7. For Fed Nest and FBO-Agg ITD, we used the same hyperparameter configuration for both the i.i.d. and non-i.i.d. settings. In particular, the inner-stepsize is 0.003, the outer-loop stepsize is 0.01, the constant λ = 0.01 and the number of inner-loop steps is 5. The choice of the number τ of outer local epochs and the data setup are indicated in the figures. Then the default value for the client participation ratio is C = 0.1. We optimize their hyperparameters via grid search guided by the default values in their source codes, to ensure the best performance given the algorithms are convergent. |