Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)
Authors: Jiayuan Ye, Reza Shokri
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In Figure 1, we observe that for data points in the first batch B0, our new privacy dynamics bound is smaller by a multiplicative factor of approximately n/b... In Figure 2 (a), we illustrate this amplification in more details. In Figure 2 (b), we compare our Rényi DP bound Theorem 4.3 with the baseline composition-based privacy bound for DP-SGD [1, 28]. We observe that for a range of RDP orders = 10, 15, 20, our privacy dynamics bound significantly improves over the baseline composition-based bound (after 50 epochs). |
| Researcher Affiliation | Academia | Jiayuan Ye, Reza Shokri Department of Computer Science National University of Singapore {jiayuan, reza}@comp.nus.edu.sg |
| Pseudocode | Yes | Algorithm 1 ANoisy-m BGD: Noisy (Stochastic) mini-batch Gradient Descent. Algorithm 2 in Appendix F.1 provides pseudocode for an equivalent of DP-SGD algorithm. |
| Open Source Code | No | The paper does not contain any explicit statements or links indicating that the source code for the described methodology is publicly available. It only refers to existing privacy libraries like TensorFlow Privacy and Opacus. |
| Open Datasets | No | The paper mentions using a 'training dataset D = (z1, ..., zn)' for image classification tasks, but does not provide specific access information (e.g., name of a public dataset like CIFAR-10/MNIST, a URL, DOI, or formal citation) to this dataset. |
| Dataset Splits | No | The paper does not provide specific details regarding training, validation, or test dataset splits (e.g., percentages, sample counts, or citations to predefined splits) that would be needed for reproduction. |
| Hardware Specification | No | The paper does not provide any specific hardware details such as GPU models, CPU types, or memory specifications used for running its experiments. |
| Software Dependencies | No | The paper mentions software like TensorFlow Privacy and Opacus but does not provide specific version numbers for these or any other ancillary software components used in the research, which are required for replication. |
| Experiment Setup | Yes | The paper provides specific experimental setup details including hyperparameters and settings for its analysis and figures. For instance, in Figures 1 and 2, it specifies: 'RDP order ε {10, 15}; λ-strongly convex loss function with λ = 1; β-smooth loss function with β = 4; gradient sensitivity Sg = 4; size of the data set n = 50; step-size = 0.02; noise variance σ2 = 4, mini-batch size b = 2.' It also defines parameters like noise multiplier σmul, regularization coefficient λ, clipping norms L and Sg2. |