Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks
Authors: Jiayuan Ye, Zhenyu Zhu, Fanghui Liu, Reza Shokri, Volkan Cevher
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To understand the relation between privacy and over-parameterization in practical DNNs training (and to validate our KL privacy bounds Lemma 3.2 and Corollary 4.2), we perform experiments for DNNs training via noisy GD to numerically estimate the KL privacy loss. We will show that if the total training time is small, it is indeed possible to obtain numerical KL privacy bound estimates that does not grow with the total number of parameter (under carefully chosen initialization distributions). Numerical estimation procedure. Theorem 3.1 proves that the exact KL privacy loss scales with the expectation of squared gradient norm during training. This could be estimated by empirically average of gradient norm across training runs. For training dataset D, we consider all car and plane images of the CIFAR-10. |
| Researcher Affiliation | Academia | National University of Singapore EPFL, Switzerland University of Warwick {jiayuan,reza}@comp.nus.edu.sg {zhenyu.zhu, volkan.cevher}@epfl.ch fanghui.liu@warwick.ac.uk |
| Pseudocode | No | The paper does not contain any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | No | The paper does not provide any concrete access to source code or explicitly state that code is made available. |
| Open Datasets | Yes | For training dataset D, we consider all car and plane images of the CIFAR-10. |
| Dataset Splits | No | The paper mentions using CIFAR-10 and creating neighboring datasets by adding/removing records, but it does not specify any training/validation/test splits with percentages or counts. |
| Hardware Specification | No | The paper does not provide any specific hardware details used for running its experiments. |
| Software Dependencies | No | The paper does not specify any software dependencies with version numbers. |
| Experiment Setup | Yes | We run noisy gradient descent with constant step-size 0.01 for 50 epochs on both datasets. ... Numerical KL privacy loss for noisy GD with constant step-size 0.001 on deep neural network with width 1024 and depth 10. ... (20 epochs, σ = 0.01, width = 1024) ... (50 epochs, σ = 0.01, width = 1024) ... (50 epochs, σ = 0.01, depth = 10). |