Hyperparameter Tuning with Renyi Differential Privacy
Authors: Nicolas Papernot, Thomas Steinke
ICLR 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To confirm these findings, we apply our algorithm to a real hyperparameter search task. Specifically, we fine-tune the learning rate of a convolutional neural network trained on MNIST. We implement DP-SGD in JAX for an all-convolutional architecture with a stack of 32, 32, 64, 64, 64 feature maps generated by 3x3 kernels. We vary the learning rate between 0.025 and 1 on a logarithmic scale but fix all other hyperparameters: 60 epochs, minibatch size of 256, ℓ2 clipping norm of 1, and noise multiplier of 1.1. In Figure 7, we plot the maximal accuracy achieved during the hyperparameter search for the different distributions considered previously as a function of the total privacy budget expended by the search. The experiment is repeated 500 times and the mean result reported. |
| Researcher Affiliation | Industry | Nicolas Papernot Google Research, Brain Team papernot@google.com Thomas Steinke Google Research, Brain Team hyper@thomas-steinke.net |
| Pseudocode | No | The paper describes its algorithm in prose and provides theorem statements, but it does not include a formally labeled "Pseudocode" or "Algorithm" block with structured steps. |
| Open Source Code | No | The paper states: "Our algorithm is simple, fully specified and can be easily implemented." However, it does not provide an explicit statement about releasing its own source code or a link to a repository. |
| Open Datasets | Yes | Specifically, we fine-tune the learning rate of a convolutional neural network trained on MNIST. |
| Dataset Splits | No | The paper mentions training on a "training set D" and evaluating quality on a "test set" or "held out set", but it does not provide specific percentages or counts for training, validation, or test splits. It implies standard usage but lacks detail for reproduction of splits. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used to run the experiments (e.g., CPU, GPU models, or memory specifications). |
| Software Dependencies | No | We implement DP-SGD in JAX for an all-convolutional architecture... The paper mentions JAX but does not specify its version number or versions of any other software dependencies. |
| Experiment Setup | Yes | We vary the learning rate between 0.025 and 1 on a logarithmic scale but fix all other hyperparameters: 60 epochs, minibatch size of 256, ℓ2 clipping norm of 1, and noise multiplier of 1.1. |