Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning
Authors: Zhekai Du, Yinjie Min, Jingjing Li, Ke Lu, Changliang Zou, Liuhua Peng, Tingjin Chu, Mingming Gong
ICLR 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments on diverse language and vision fine-tuning tasks demonstrate that Lo CA offers enhanced parameter efficiency while maintains computational feasibility comparable to low-rank-based methods. |
| Researcher Affiliation | Academia | University of Electronic Science and Technology of China The University of Melbourne Nankai University Mohamed bin Zayed University of Artificial Intelligence |
| Pseudocode | Yes | Algorithm 1 Lo CA Fine-tuning |
| Open Source Code | No | We implement our method using the Py Torch framework. Our code is built on the PEFT library (Mangrulkar et al., 2022) from Huggingface, and all pre-trained models are sourced from Huggingface s Transformers library (Wolf et al., 2020). |
| Open Datasets | Yes | Alpaca-52K dataset (Taori et al., 2023), GLUE benchmark (Wang et al., 2018), E2E NLG Challenge dataset (Novikova et al., 2017), MT-Bench (Zheng et al., 2024) and Vicuna (Chiang et al., 2023) datasets, Oxford Pets (Parkhi et al., 2012), Stanford Cars (Krause et al., 2013), CIFAR10 (Krizhevsky et al., 2009), DTD (Cimpoi et al., 2014), Euro SAT (Helber et al., 2019), FGVC (Maji et al., 2013), RESISC45 (Cheng et al., 2017) and CIFAR100 (Krizhevsky et al., 2009). |
| Dataset Splits | Yes | We report the best results on the validation set for each task. Mean results are reported after 3 runs with different random seeds. ... We evaluate the model on the MT-Bench (Zheng et al., 2024) and Vicuna (Chiang et al., 2023) datasets ... All results are obtained after 5 random trials. |
| Hardware Specification | Yes | All PEFT experiments are conducted on a single NVIDIA Tesla H100 GPU. |
| Software Dependencies | No | We implement our method using the Py Torch framework. Our code is built on the PEFT library (Mangrulkar et al., 2022) from Huggingface, and all pre-trained models are sourced from Huggingface s Transformers library (Wolf et al., 2020). |
| Experiment Setup | Yes | For the alternating optimization, we used Ba = 10 and Bl = 20. The coefficients a are initialized to be zeros and the locations l are randomly initialized with a uniform distribution. ... Detailed hyperparameters can be found in Table 6. (Tables 6, 7, 8, and 9 provide specific hyperparameter values like learning rates, batch sizes, epochs, weight decay, etc.) |