Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
When Does Differentially Private Learning Not Suffer in High Dimensions?
Authors: Xuechen Li, Daogao Liu, Tatsunori B. Hashimoto, Huseyin A. Inan, Janardhan Kulkarni, Yin-Tat Lee, Abhradeep Guha Thakurta
NeurIPS 2022 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically show that in private fine-tuning of large language models, gradients obtained during fine-tuning are mostly controlled by a few principal components. |
| Researcher Affiliation | Collaboration | Xuechen Li Stanford University, Daogao Liu University of Washington, Tatsunori Hashimoto Stanford University, Huseyin A. Inan Microsoft Research, Janardhan Kulkarni Microsoft Research, Yin Tat Lee University of Washington Microsoft Research, Abhradeep Guha Thakurta Google Research |
| Pseudocode | Yes | Algorithm 1 presents the pseudocode. |
| Open Source Code | Yes | Code to reproduce our results can be found at https://github.com/lxuechen/private-transformers/tree/main/examples/classification/spectral_analysis. |
| Open Datasets | Yes | pretrained BERT [DCLT18] and GPT-2 [RNSS18, RWC+19] models can be fine-tuned... Res Nets [HZRS16] and vision-Transformers [DBK+20] can be fine-tuned to perform well for Image Net classification... Distil Ro BERTa [SDCW19, LOG+19] under ε = 8 and δ = 1/n1.1 for sentiment classification on the SST-2 dataset [SPW+13]. |
| Dataset Splits | Yes | We include additional experimental setup details in Appendix C... which were tuned on separate validation data. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory) used for running its experiments. It only refers to 'large pretrained models' and 'fine-tuning gigantic parameter vectors'. |
| Software Dependencies | No | The paper mentions 'pretrained BERT', 'GPT-2', 'Res Nets', 'vision-Transformers', 'Distil Ro BERTa' as models, but does not specify software dependencies with version numbers (e.g., PyTorch 1.9, Python 3.8). |
| Experiment Setup | Yes | Specifically, we finetune Distil Ro BERTa [SDCW19, LOG+19] under ε = 8 and δ = 1/n1.1 for sentiment classification on the SST-2 dataset [SPW+13]. We reformulate the label prediction problem as templated text prediction [LTLH21], and fine-tune only the query and value matrices in attention layers... we over-train by privately fine-tuning for r = 2 × 10^3 updates. |