HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
Authors: Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, Cheng-Zhong Xu
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments demonstrate that Hydra Lo RA outperforms other PEFT approaches, even those that rely on domain knowledge during the training and inference phases. Code is available. 4 Experiments In this section, we detail the principal experiments. We begin with an overview of the experimental setup and implementation intricacies. Following this, we share our findings and offer a succinct interpretation. |
| Researcher Affiliation | Academia | Chunlin Tian University of Macau yc27402@um.edu.mo Zhan Shi University of Texas at Austin zshi17@cs.utexas.edu Zhijiang Guo University of Cambridge zg283@cam.ac.uk Li Li* University of Macau llili@um.edu.mo Chengzhong Xu University of Macau czxu@um.edu.mo |
| Pseudocode | No | No pseudocode or algorithm blocks are explicitly labeled or presented in a structured format. |
| Open Source Code | Yes | Code is available. We provide a detailed explanation of parameter usage and will release the source code to ensure reproducibility. This paper relates the details of the code as part of the submission. |
| Open Datasets | Yes | General: we fine-tune with the general instruction tuning databricks-dolly-15k [8] for generic language capability and evaluate with MMLU [16]. Medical: we fine-tune with Gen Med GPT and clinic-10k from Chat Doctor [26] for medicine applications and evaluate medical tasks in MMLU. |
| Dataset Splits | Yes | Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: The full details are provided in the appendix and supplemental material (code). |
| Hardware Specification | Yes | The following experiments were executed on a GPU infrastructure consisting of 4 NVIDIA A40 GPUs and a CPU powered by an Intel(R) Xeon(R) Gold 6330 CPU clocked at 2.00GHz. |
| Software Dependencies | No | No specific software dependencies with version numbers are mentioned in the paper. |
| Experiment Setup | Yes | To explore the properties and commonalities of the Lo RA asymmetric structure, we conduct experiments on both single and multiple domains to evaluate the effectiveness of Hydra Lo RA for profiling intrinsic components. 8-shot for GSM8K, zero-shot for others. For Lo RA (Split) decomposes highrank Lo RA modules into smaller, equivalent lowrank components (r n). n is the number of Lo RAs, r denotes the rank of each Lo RA. |