Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification
Authors: Yingxue Zhou, Steven Wu, Arindam Banerjee
ICLR 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically evaluate PDP-SGD on training neural networks with two datasets: the MNIST (Le Cun et al., 1998) and Fashion MNIST (Xiao et al., 2017). We compare the performance of PDP-SGD with the baseline DP-SGD for various privacy levels ϵ. In addition, we also explore a heuristic method, i.e., DP-SGD with random projection by replacing the projector with a Rk p Gaussian random projector (Bingham and Mannila, 2001; Blocki et al., 2012). We present the experimental results after discussing the experimental setup. More details and additional results are in Appendix D. |
| Researcher Affiliation | Academia | Yingxue Zhou , Zhiwei Steven Wu , Arindam Banerjee Department of Computer Science & Engineering, University of Minnesota School of Computer Science, Carnegie Mellon University. Department of Computer Science, University of Illinois Urbana-Champaign |
| Pseudocode | Yes | Algorithm 1 Projected DP-SGD (PDP-SGD) |
| Open Source Code | No | The paper does not explicitly provide access to the source code for the methodology described in the paper. |
| Open Datasets | Yes | We empirically evaluate PDP-SGD on training neural networks with two datasets: the MNIST (Le Cun et al., 1998) and Fashion MNIST (Xiao et al., 2017). |
| Dataset Splits | No | The paper describes training and test sets and a small public dataset but does not explicitly provide details about a separate validation dataset split. |
| Hardware Specification | Yes | All experiments have been run on NVIDIA Tesla K40 GPUs. |
| Software Dependencies | No | The paper mentions 'Py Torch' but does not specify version numbers for any software dependencies. |
| Experiment Setup | Yes | The mini-batch size is set to be 250 for both MNIST and Fashion MNIST. For the step size, we follow the grid search method with search space {0.01, 0.05, 0.1, 0.2} to tune the step size for MNIST and the search space is {0.01, 0.02, 0.05, 0.1, 0.2} for Fashion MNIST. We choose the step size based on the training accuracy at the last epoch. [...] For training, a fixed budget on the number of epochs i.e., 30 is assigned for the each task. [...] We choose gradient clip size to be 1.0 for both datasets. |