Self-Ensemble Protection: Training Checkpoints Are Good Data Protectors
Authors: Sizhe Chen, Geng Yuan, Xinwen Cheng, Yifan Gong, Minghai Qin, Yanzhi Wang, Xiaolin Huang
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 4 EXPERIMENTS We evaluate SEP along with 7 data protection baselines, including adding random noise, Tensor Clog aiming to cause gradient vanishing (Shen et al., 2019), Gradient Alignment to target-class gradients (Fowl et al., 2021a), Deep Confuse that protects by an autoencoder (Feng et al., 2019), Unlearnable Examples (ULEs) using error-minimization noise (Huang et al., 2020a), Robust ULEs (RULEs) that use adversarial training (Fu et al., 2021), Adversarial Poison (Adv Poison) resorting to targeted attacks (Fowl et al., 2021b), and Auto Regressive (AR) Poison (Sandoval-Segura et al., 2022) using Markov chain. |
| Researcher Affiliation | Academia | Sizhe Chen1,2, Geng Yuan2, Xinwen Cheng1, Yifan Gong2, Minghai Qin2, Yanzhi Wang2, Xiaolin Huang1 1Department of Automation, Shanghai Jiao Tong University 2Department of Electrical and Computer Engineering, Northeastern University |
| Pseudocode | Yes | Algorithm 1 Self-Ensemble Protection with Feature Alignment and Variance Reduction |
| Open Source Code | Yes | Code is available at https://github.com/Sizhe-Chen/SEP. |
| Open Datasets | Yes | For our method, we optimize class-y samples to have the mean feature of target incorrect class g(y), where g(y) = (y + 5)%10 for CIFAR-10 (Krizhevsky et al., 2009) and Image Net (Krizhevsky et al., 2017) protected classes, and g(y) = (y + 50)%100 for CIFAR-100. |
| Dataset Splits | Yes | The validation (using 2500 samples separated from training data) performance of different types of perturbations by our method (CIFAR-10, ℓ = 8/255, ℓ2 = 1, fp = fa = Res Net18). |
| Hardware Specification | Yes | Experiments are conducted on an NVIDIA Tesla A100 GPU but could be run on GPUs with 4GB+ memory because we store checkpoints on hardware. |
| Software Dependencies | No | They are implemented from Pytorch vision (Paszke et al., 2019). (Does not provide version numbers for PyTorch or other software dependencies). |
| Experiment Setup | Yes | We train a Res Net18 for N = 120 epochs as fp following (Huang et al., 2020a; Fowl et al., 2021b). 15 equidistant intermediate checkpoints (epoch 8, 16, ..., 120) are adopted with M = 15, T = 30 if not otherwise stated. (...) We train appropriator DNNs fa for 120 epochs by an SGD optimizer with an initial learning rate of 0.1, which is divided by 10 in epochs 75 and 90. The momentum item in training is 0.9 and the weight decay is 5e-4. |