DoRA: Weight-Decomposed Low-Rank Adaptation
Authors: Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Do RA consistently outperforms Lo RA on fine-tuning LLa MA, LLa VA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding. |
| Researcher Affiliation | Collaboration | 1NVIDIA 2HKUST. |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code is available at https://github.com/ NVlabs/Do RA. |
| Open Datasets | Yes | The commonsense reasoning tasks comprise 8 sub-tasks, each with a predefined training and testing set. We follow the setting of (Hu et al., 2023) and amalgamate the training datasets from all 8 tasks to create the final training dataset and conduct evaluations on the individual testing dataset for each task. ... The training datasets contain several datasets from VQA (Goyal et al., 2017; Hudson & Manning, 2019; Marino et al., 2019; Schwenk et al., 2022), OCR (Mishra et al., 2019; Sidorov et al., 2020), region-level VQA (Kazemzadeh et al., 2014; Krishna et al., 2017; Mao et al., 2016), visual conversation (Liu et al., 2023a), and language conversation data. |
| Dataset Splits | No | The commonsense reasoning tasks comprise 8 sub-tasks, each with a predefined training and testing set. We follow the setting of (Hu et al., 2023) and amalgamate the training datasets from all 8 tasks to create the final training dataset and conduct evaluations on the individual testing dataset for each task. The paper does not explicitly state the validation split percentages or counts. |
| Hardware Specification | No | We conduct an ablation study to evaluate the impact of the proposed modification on fine-tuning LLa MA-7B and VL-BART. The results indicate that the modification leads to a training memory reduction of approximately 24.4% in fine-tuning LLa MA and 12.4% in VL-BART. Furthermore, the accuracy of Do RA with the modification remains unchanged for VL-BART and shows a negligible difference of only 0.2 compared to Do RA without the modification on LLa MA. (This only refers to "training memory reduction" and "GPU Memory Cost (GB)" in Table 7, but no specific hardware models.) |
| Software Dependencies | No | The paper does not specify software dependencies with version numbers. |
| Experiment Setup | Yes | Table 8. Hyperparameter configurations of Do RA for LLa MA-7B/13B, LLa MA2-7B, and LLa MA3-8B on the commonsense reasoning tasks. ... Table 9. Hyperparameter configurations of Do RA for fine-tuning VL-Bart on image/video-text tasks. ... Table 10. Hyperparameter configurations of Do RA and Lo RA for fine-tuning LLa VA-1.5-7B with visual instruction tuning datasets. ... Table 11. Hyperparameter configurations of Do RA and DVo RA for fine-tuning LLa MA-7B and LLa MA2-7B with cleaned Alpaca dataset. |