Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
GoRA: Gradient-driven Adaptive Low Rank Adaptation
Authors: haonan he, Peng Ye, Yuchen Ren, yuan yuan, LuyangZhou, ShucunJu, lei chen
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments across various architectures and modalities show that Go RA consistently outperforms existing Lo RA-based methods while preserving the efficiency of vanilla Lo RA. For example, when fine-tuning Llama3.1-8B-Base for mathematical reasoning, Go RA achieves a 5.13-point improvement over standard Lo RA and even outperforms full fine-tuning by 2.05 points under high-rank settings. |
| Researcher Affiliation | Academia | Haonan He1,2,3 , Peng Ye3,4,5 , Yuchen Ren3,6, Yuan Yuan2, Luyang Zhou7, Shucun Ju7, Lei Chen2 1University of Science and Technology of China 2Institute of Intelligent Machines, HFIPS, Chinese Academy of Sciences 3Shanghai Artificial Intelligence Laboratory 4Fudan University 5The Chinese University of Hong Kong 6University of Sydney 7Anhui Disaster Warning & Agrometeorological Information Center EMAIL |
| Pseudocode | Yes | Algorithm 1 Rank Allocation and Initialization of Go RA under Single Training Worker |
| Open Source Code | Yes | Code is available at: https://github.com/hhnqqq/My Transformers. |
| Open Datasets | Yes | For understanding tasks, we trained T5-Base [32] on five tasks of GLUE [33] (MNLI, SST-2, Co LA, QNLI, MRPC) and reported accuracy on corresponding validation sets. For generation tasks, we fine-tuned Llama-3.1-8B-Base [2] and Llama-2-7B-Base [1] on chat, mathematics, and coding datasets, evaluating test performance on MTBench [34], GSM8k [35], and Human Eval [36]. For image classification tasks, we fine-tuned CLIP-Vi T-B/16 [37] on seven datasets including Stanford Cars [38], DTD [39], Euro SAT [40], GT-SRB [41], RESISC45 [42], SUN397 [43] and SVHN [44] and reported test accuracy. |
| Dataset Splits | Yes | For understanding tasks, we trained T5-Base [32] on five tasks of GLUE [33] (MNLI, SST-2, Co LA, QNLI, MRPC) and reported accuracy on corresponding validation sets. For generation tasks, we fine-tuned Llama-3.1-8B-Base [2] and Llama-2-7B-Base [1] on chat, mathematics, and coding datasets, evaluating test performance on MTBench [34], GSM8k [35], and Human Eval [36]. |
| Hardware Specification | Yes | For natural language understanding tasks reported in section 4.1, we conduct our experiments using the Huggingface Transformers framework for model and trainer implementation on a single RTX 4090 24GB. In contrast, for natural language generation tasks reported in Section 4.2 and Section 5, we utilize the Deep Speed Ze RO2 [25] data parallel framework and Flash Attention-2 [58] mechanism, leveraging the power of 8 RTX 4090 24 GB GPUs or 8 A800 80 GB GPUs. |
| Software Dependencies | No | For natural language understanding tasks reported in section 4.1, we conduct our experiments using the Huggingface Transformers framework for model and trainer implementation on a single RTX 4090 24GB. In contrast, for natural language generation tasks reported in Section 4.2 and Section 5, we utilize the Deep Speed Ze RO2 [25] data parallel framework and Flash Attention-2 [58] mechanism, leveraging the power of 8 RTX 4090 24 GB GPUs or 8 A800 80 GB GPUs. All codes of Go RA and baseline methods are implemented in Py Torch. |
| Experiment Setup | Yes | Unless specified otherwise, we set the Lo RA rank or Go RA s reference rank rref to 8. The hyperparameters of Go RA are detailed in Appendix C.3. Settings: We adopted baseline performances reported by Lo RA-GA [18], maintaining their experimental parameters for fair comparison: Adam[7] optimizer (β1 = 0.9, β2 = 0.999, weight decay = 0), batch size 32, cosine decay learning rate with a warmup ratioo of 0.03. We trained all linear layers except the language head using a peak learning rate of 1e-4, a maximum sequence length of 128, and FP32 precision. |