Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation
Authors: Yi-Chen Li, Fuxiang Zhang, Wenjie Qiu, Lei Yuan, Chengxing Jia, Zongzhang Zhang, Yang Yu, Bo An
ICLR 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments based on the Llama-3.1 model on the DSP dataset and HH-RLHF dataset illustrate the superior effectiveness of QAdapter on both retaining existing knowledge and learning new preferences. Our code is available at https://github.com/LAMDA-RL/Q-Adapter. |
| Researcher Affiliation | Collaboration | 1 National Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China, 2 School of Artificial Intelligence, Nanjing University, Nanjing, China, 3 Nanyang Technological University, Singapore, 4 Polixir Technologies, Nanjing, China, 5 Skywork AI, Singapore EMAIL, EMAIL, EMAIL, EMAIL, EMAIL |
| Pseudocode | Yes | Algorithm 1 Q-Adapter |
| Open Source Code | Yes | Our code is available at https://github.com/LAMDA-RL/Q-Adapter. |
| Open Datasets | Yes | Experiments based on the Llama-3.1 model on the DSP dataset and HH-RLHF dataset illustrate the superior effectiveness of QAdapter on both retaining existing knowledge and learning new preferences. |
| Dataset Splits | No | We choose the Domain Specific Preference (DSP) (Cheng et al., 2023) dataset that contains preference data from four domains including academy, business, entertainment and literature. For the second one, we choose the HH-RLHF (Bai et al., 2022) dataset that has two folds of preference data: helpful and harmless. The helpful data will first be used to post-train a Llama-3.1 model. Then we will use the harmless data to further customize it. |
| Hardware Specification | Yes | We train QAdapter in one single machine with 4x NVIDIA Ge Force RTX 4090 GPUs. |
| Software Dependencies | No | We adopt the peft library (Mangrulkar et al., 2022) to modify the linear layers of the model with a rank of 8. We adopt the Hugging Face trainer to control the training procedure with a rewritten loss function. We use the TRL (von Werra et al., 2020) implementation of the DPO trainer to train our DPO model with Lo RA. |
| Experiment Setup | Yes | In Table 3, we list the basic hyper-parameters of Q-Adapter. Hyper-parameter Value α 1.0 α0 0.01 β 0.1 γ 1.0 optimizer Adam W learning rate 3e-4 batch size 512 max sequence length T 512 |