Privacy-Preserving Instructions for Aligning Large Language Models
Authors: Da Yu, Peter Kairouz, Sewoong Oh, Zheng Xu
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In both supervised fine-tuning and reinforcement learning from human feedback, our extensive experiments demonstrate the high utility of the final set of synthetic instructions by showing comparable results to real instructions. |
| Researcher Affiliation | Collaboration | Da Yu Sun Yat-sen University, the work was done when Da Yu was an intern at Google Research. yuda3@mail2.sysu.edu.cn. Google Research, alphabetical order. {kairouz,sewoongo,xuzheng}@google.com. |
| Pseudocode | Yes | Algorithm 1 Train DP Generator to Synthesize Instructions. Algorithm 2 Resample Synthetic Data with DP Histogram. |
| Open Source Code | Yes | The code to reproduce the main findings in our paper is available at: https://github.com/google-research/google-research/tree/master/dp_instructions. |
| Open Datasets | Yes | We use the LMSYS-Chat-1M dataset as the private dataset (Zheng et al., 2023a)...For this, we take instructions from the FLAN dataset (Chung et al., 2022; Longpre et al., 2023). |
| Dataset Splits | Yes | The Chatbot Arena instructions are then divided into three subsets: 180,000 for the training set, 5,000 for the validation set, and the rest for the test set. |
| Hardware Specification | Yes | Our experiments use 32 Nvidia A100 40G GPUs. |
| Software Dependencies | No | The paper mentions several software components and libraries (e.g., LLaMA models, LoRA, DP-Adam, Adam optimizer, Sentence-T5-base, Faiss package, GPT-3.5-Turbo, GPT-4, phi-1.5, TRL library) but does not provide specific version numbers for them. |
| Experiment Setup | Yes | Our hyperparameter sweeps for DP fine-tuning and non-private fine-tuning are outlined in Table 9 and Table 10, respectively...The KL penalty coefficient is set as 0.2 and the number of optimisation steps per batch is 4. We use a batchsize of 128 and a learning rate of 1.41 × 10−5. |