Better Diffusion Models Further Improve Adversarial Training
Authors: Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, Shuicheng Yan
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our adversarially trained models achieve state-of-the-art performance on Robust Bench using only generated data (no external datasets). Under the ℓ norm threat model with ϵ = 8/255, our models achieve 70.69% and 42.67% robust accuracy on CIFAR-10 and CIFAR-100, respectively, i.e. improving upon previous state-of-the-art models by +4.58% and +8.03%. These results also beat previous works that use external data. We also provide compelling results on the SVHN and Tiny Image Net datasets. Our code is at https://github.com/wzekai99/DM-Improves-AT. We conduct extensive ablation studies to better reveal the mechanism by which diffusion models promote the AT process. |
| Researcher Affiliation | Collaboration | Zekai Wang * 1 Tianyu Pang * 2 Chao Du 2 Min Lin 2 Weiwei Liu 1 Shuicheng Yan 2. 1School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University. 2Sea AI Lab. |
| Pseudocode | No | The paper does not contain any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | Yes | Our code is at https://github.com/wzekai99/DM-Improves-AT. |
| Open Datasets | Yes | CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009) consist of 50K training images and 10K test images with 10 and 100 classes, respectively. All CIFAR images are 32 32 3 resolution (width, height, RGB channel). SVHN (Netzer et al., 2011) contains 73,257 training and 26,032 test images (0 9 small cropped digits, 10 classes). Tiny Image Net4 contains 100K images for training, and 10K images for testing. |
| Dataset Splits | Yes | We separate first 1024 images of training set as a fixed validation set. During every epoch of AT, we pick the best checkpoint by evaluating robust accuracy under PGD-40 attack on the validation set. |
| Hardware Specification | Yes | The experiments on WRN-28-10 are parallelly processed with four NVIDIA A100 SXM4 40GB GPUs. To evaluate how the abundant generated data affects large networks, we further use WRN-70-16 in Section 4, which contains 267M parameters. We use eight A100 GPUs to train WRN-70-16. |
| Software Dependencies | No | The paper mentions 'Py Torch implementation' but does not specify version numbers for PyTorch or any other software dependencies. |
| Experiment Setup | Yes | We follow the basic setup and use the Py Torch implementation of Rebuffi et al. (2021). Training settings: We use TRADES (Zhang et al., 2019b) as the framework of adversarial training (AT), with β = 5 for CIFAR-10/CIFAR-100, β = 6 for SVHN, and β = 8 for Tiny Image Net. We adopt weight averaging with decay rate τ = 0.995 (Izmailov et al., 2018). We use the SGD optimizer with Nesterov momentum (Nesterov, 1983), where the momentum factor and weight decay are set to 0.9 and 5 10 4, respectively. We use the cyclic learning rate schedule with cosine annealing (Smith & Topin, 2019), where the initial learning rate is set to 0.2. In Section 6, we test the sensitivity of basic training hyperparameters on CIFAR-10. WRN-28-10 models are trained for 400 epochs using 1M data generated by EDM. 512 is the default batch size unless otherwise specified. |