BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models
Authors: Yibin Wang, Haizhou Shi, Ligong Han, Dimitris Metaxas, Hao Wang
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our empirical results verify the effectiveness of BLo B in terms of generalization and uncertainty estimation, when evaluated on both in-distribution and out-of-distribution data. |
| Researcher Affiliation | Collaboration | 1Rutgers University. 2MIT-IBM Watson AI Lab. |
| Pseudocode | Yes | Algorithm 1 Bayesian Low-Rank Adaptation by Backpropagation (BLo B) |
| Open Source Code | Yes | Code is available at https://github.com/Wang-ML-Lab/bayesian-peft. |
| Open Datasets | Yes | We fine-tune Llama2-7B on six common-sense reasoning tasks: Winogrande-small (WG-S), Winogrande-medium (WG-M) [82], ARC-Challenge (ARC-C) [18], ARC-Easy (ARC-E) [18], Open Book QA (OBQA) [65], and Bool Q [17]. |
| Dataset Splits | Yes | For all baseline methods, using the same pre-trained LLM backbone, we maintain consistent hyperparameters across all datasets and do not use additional validation sets to achieve higher performance (See Appendix B.3 for detailed settings). |
| Hardware Specification | Yes | Our experiments on Llama2-7B were conducted using 2 NVIDIA RTX A5000 GPUs for parallel training, while experiments on Ro BERTa-base were conducted using 4 NVIDIA RTX A5000 GPUs for parallel training. |
| Software Dependencies | No | The paper mentions the use of specific software libraries like "PEFT library" and "Bayesian-Torch library" but does not provide specific version numbers for these dependencies within the main text or appendices to ensure reproducibility. |
| Experiment Setup | Yes | For hyperparameters, we strictly adhere to the default settings in the PEFT library and the original Lo RA paper [63, 41] to ensure maximal reproducibility. This includes the number of training steps, learning rate, and Lo RA rank r (see Appendix B.1 for details). ... Detailed hyperparameter settings are provided in the Table 4. Table 3 provides the hyperparameters for fine-tuning with Lo RA shared with other baselines. |