Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA
Authors: Shuangyi Chen, Yuanxin Guo, Yue Ju, Hardik Dalal, Zhongwen Zhu, Ashish Khisti
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We use both theoretical analysis and extensive experiments to demonstrate the advantages of Ro Lo RA over prior approaches... To bridge theory and practice, we conducted extensive experimental evaluations on language models including Ro BERTa-Large, Llama-2-7B on diverse tasks and FL settings to demonstrate the advantages of Ro Lo RA over other methods. |
| Researcher Affiliation | Collaboration | Shuangyi Chen1 University of Toronto EMAIL, Yue Ju Ericsson-GAIA Montréal EMAIL |
| Pseudocode | Yes | Algorithm 1 Ro Lo RA iterations and Algorithm 2 Ro Lo RA for linear regressor, Alt-min-GD iterations in Appendix A1. |
| Open Source Code | Yes | The datasets are all open-source. The code is uploaded. |
| Open Datasets | Yes | We train the model on MNIST [11]. We take the pre-trained Ro BERTa-Large (355M) [27] models from the Hugging Face Transformers library, and evaluate the performance of federated finetuning methods on 5 datasets (SST-2, QNLI, MNLI, QQP, RTE) from the GLUE [38]. Code Alpaca [4] for coding tasks, and Alpaca [33] for general instruction-following tasks. Using Human Eval [5] as the metric for Code Alpaca, we assess the model s ability to generate accurate code solutions. For Alpaca, we employ MMLU (Massive Multitask Language Understanding) [18]. |
| Dataset Splits | Yes | We consider two different ways to distribute training images to clients. The first is to distribute the images to 5 clients and each client gets access to training images of two specific labels, while the second is to distribute the images to 10 clients and each client only has training images of one specific label. There is no overlap in the training samples each client can access. In Table 1, we increased the number of clients from 3 to 20, and then to 50, ensuring that there is no overlap in the training samples each client can access. Dirichlet(0.5) with 10 clients and Dirichlet(1.0) with 15 clients. |
| Hardware Specification | Yes | We use NVIDIA Ge Force RTX 4090 or NVIDIA A40 for all the experiments. |
| Software Dependencies | No | We implement all the methods based on Federated Scope-LLM [24] and We take the pre-trained Ro BERTa-Large (355M) [27] models from the Hugging Face Transformers library. No specific version numbers for these software components are provided. |
| Experiment Setup | Yes | Specifically, the learning rate is chosen from the set {5e 4, 1e 3, 2e 3, 5e 3, 1e 2, 2e 2, 5e 2, 1e 1}. Other hyper-parameters for experiments are specified in Table 6 in Appendix A5.2. Table 6 includes 'Total comm. rounds', 'Batch Size', 'Local Epochs'. |