FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations
Authors: Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, Ang Li
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments demonstrate FLORA s superior performance in both homogeneous and heterogeneous settings, surpassing stateof-the-art methods. |
| Researcher Affiliation | Collaboration | University of Maryland, College Park 2. Rutgers University 3. Gen Bio.ai 4. Sony Reasearch |
| Pseudocode | No | The paper includes diagrams and mathematical equations to describe the method, but does not present a formal pseudocode or algorithm block. |
| Open Source Code | Yes | Our code is available at https://github.com/ATP-1010/Federated LLM. |
| Open Datasets | Yes | We use the Databricks-dolly-15k [28] instruction dataset, Alpaca dataset [19], and Wizard dataset [14] for the question-answering (QA) task, and Wizard and Share GPT for the chat assistant task. |
| Dataset Splits | No | The paper states the datasets used for training (Dolly, Alpaca, Wizard, Share GPT) and evaluation (MMLU, MT-bench), and mentions a non-IID client sampling setting. However, it does not provide explicit train/validation/test dataset splits (e.g., percentages or counts) for the training data or specify how a validation set was derived or used. |
| Hardware Specification | Yes | We use a 256GB AMD EPYC 7763 64-Core Processor on Linux v4.18.0 to run the experiments. For Lo RA fine-tuning on all the models, we use 4 NVIDIA RTX A6000 GPUs. |
| Software Dependencies | No | The paper does not explicitly list software dependencies with specific version numbers (e.g., Python 3.x, PyTorch 1.x). |
| Experiment Setup | Yes | In all our experiments, the learning rate of fine-tuning is set to 0.0003; the batch size is 128 and the micro batch size is 16. Table 2 shows the fine-tuning rounds and local epochs we selected. |