Federated Model Heterogeneous Matryoshka Representation Learning
Authors: Liping Yi, Han Yu, Chao Ren, Gang Wang, xiaoguang Liu, Xiaoxiao Li
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments on benchmark datasets demonstrate its superior model accuracy with low communication and computational costs compared to seven state-of-the-art baselines. It achieves up to 8.48% and 24.94% accuracy improvement compared with the state-of-the-art and the best same-category baseline, respectively. |
| Researcher Affiliation | Academia | Liping Yi1,2, Han Yu2, Chao Ren2, Gang Wang1, , Xiaoguang Liu1, , Xiaoxiao Li3,4 1College of Computer Science, TMCC, Sys Net, DISSec, GTIISC, Nankai University, China 2College of Computing and Data Science, Nanyang Technological University, Singapore 3Department of Electrical and Computer Engineering, The University of British Columbia, Canada 4Vector Institute, Canada {yiliping, wgzwp, liuxg}@nbjl.nankai.edu.cn {han.yu, chao.ren}@ntu.edu.sg, xiaoxiao.li@ece.ubc.ca |
| Pseudocode | Yes | Algorithm 1: Fed MRL Input: N, total number of clients; K, number of selected clients in one round; T, total number of rounds; ηω, learning rate of client local heterogeneous models; ηθ, learning rate of homogeneous small model; ηφ, learning rate of the representation projector. Output: client whole models removing the global header [G(θex,T 1) F0(ωT 1 0 )|P0(φT 1 0 ), . . . , G(θex,T 1) FN 1(ωT 1 N 1)|PN 1(φT 1 N 1)]. |
| Open Source Code | Yes | 4https://github.com/Liping Yi/Fed MRL |
| Open Datasets | Yes | The benchmark datasets adopted are CIFAR-10 and CIFAR-100 5 [23], which are commonly used in FL image classification tasks for the evaluating existing MHetero FL algorithms. CIFAR-10 has 60, 000 32 32 colour images across 10 classes, with 50, 000 for training and 10, 000 for testing. CIFAR-100 has 60, 000 32 32 colour images across 100 classes, with 50, 000 for training and 10, 000 for testing. We follow [40] and [37] to construct two types of non-IID datasets. Each client s non-IID data are further divided into a training set and a testing set with a ratio of 8 : 2. |
| Dataset Splits | Yes | Each client s non-IID data are further divided into a training set and a testing set with a ratio of 8 : 2. |
| Hardware Specification | Yes | The experiments are carried out over two benchmark supervised image classification datasets on 4 NVIDIA Ge Force 3090 GPUs (24GB Memory). |
| Software Dependencies | No | We implement Fed MRL on Pytorch, and compare it with seven state-of-the-art MHetero FL methods. The paper mentions Pytorch but does not provide a specific version number or list other software dependencies with version numbers. |
| Experiment Setup | Yes | We search optimal FL hyperparameters and unique hyperparameters for all MHetero FL algorithms. For FL hyperparameters, we test MHetero FL algorithms with a {64, 128, 256, 512} batch size, {1, 10} epochs, T = {100, 500} communication rounds and an SGD optimizer with a 0.01 learning rate. The unique hyperparameter of Fed MRL is the representation dimension d1 of the homogeneous global small model, we vary d1 = {100, 150, ..., 500} to obtain the best-performing Fed MRL. |