Public Data-Assisted Mirror Descent for Private Model Training
Authors: Ehsan Amid, Arun Ganesh, Rajiv Mathews, Swaroop Ramaswamy, Shuang Song, Thomas Steinke, Thomas Steinke, Vinith M Suriyakumar, Om Thakkar, Abhradeep Thakurta
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the efficacy of our algorithm by showing privacy/utility trade-offs on four benchmark datasets (Stack Overflow, Wiki Text-2, CIFAR-10, and EMNIST). |
| Researcher Affiliation | Collaboration | 1Google 2UC Berkeley 3Part of this work was done while the author was an intern at Google. 4MIT. |
| Pseudocode | Yes | Algorithm 1 Public Data-Assisted Differentially Private Mirror Descent (PDA-DPMD) |
| Open Source Code | Yes | The code has been open sourced. |
| Open Datasets | Yes | Our empirical results are either on simulated linear regression data, or on standard benchmark datasets like Stack Overflow, CIFAR-10, EMNIST, and Wiki Text-2 |
| Dataset Splits | Yes | For each dataset, we randomly assign 4% of the original training data as public, and the rest as private. |
| Hardware Specification | No | The paper mentions "a Tesla V100 GPU" in the context of trying to run another work's implementation, but does not provide specific hardware details (like model numbers, processors, or memory) for its own experimental setup. |
| Software Dependencies | No | The paper mentions using specific models like "one layer LSTM from (Kairouz et al., 2021b)" but does not specify software dependencies with version numbers (e.g., PyTorch 1.9, TensorFlow 2.x, CUDA 11.1). |
| Experiment Setup | Yes | We fix the noise multiplier to σ = 0.4 so that the client-level ε = 8.32 for δ = 10−6. We perform a grid search over the server learning rate {0.5, 1.0, 3.0}, the client learning rate {0.1, 0.2, 0.5}, and clipping norm {0.3, 1.0} for both methods. We perform an additional search for the optimal decay schedule for α for PDA-DPMD using cos(πt/(iT)) for i ∈ {2, 3, 4, 5, 8}, where T = 1600. |