Consistency Regularization for Adversarial Robustness
Authors: Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minseon Kim, Sung Ju Hwang, Jinwoo Shin8414-8422
AAAI 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experimental results demonstrate that such a simple regularization technique brings significant improvements in the test robust accuracy of a wide range of AT methods. More remarkably, we also show that our method could significantly help the model to generalize its robustness against unseen adversaries, e.g., other types or larger perturbations compared to those used during training. Code is available at https://github.com/alinlab/consistency-adversarial. ... Overall, our experimental results show that the proposed regularization can be easily adapted for a wide range of AT methods to prevent overfitting in robustness. ... Experiments We verify the effectiveness of our technique on image classification datasets: CIFAR-10/100 (Krizhevsky and Hinton 2009) and Tiny-Image Net. Our results exhibit that incorporating simple consistency regularization scheme into the existing adversarial training (AT) methods significantly improve adversarial robustness against various attacks (Carlini and Wagner 2017; Madry et al. 2018; Croce and Hein 2020b), including data corruption (Hendrycks and Dietterich 2019). Intriguingly, our method shows better robustness against unseen adversaries compared to other baselines. Moreover, our method surpass the performance of the recent regularization technique (Wu, Xia, and Wang 2020). Finally, we perform an ablation study to validate each component of our approach. |
| Researcher Affiliation | Collaboration | Jihoon Tack1, Sihyun Yu1, Jongheon Jeong1, Minseon Kim1, Sung Ju Hwang1,2, Jinwoo Shin1 1Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea 2AITRICS, Seoul, South Korea {jihoontack,sihyun.yu,jongheonj,minseonkim,sjhwang82,jinwoos}@kaist.ac.kr |
| Pseudocode | No | The paper does not contain any clearly labeled pseudocode or algorithm blocks. It describes the methods using mathematical formulas and textual descriptions. |
| Open Source Code | Yes | Code is available at https://github.com/alinlab/consistency-adversarial. |
| Open Datasets | Yes | We verify the effectiveness of our technique on image classification datasets: CIFAR-10/100 (Krizhevsky and Hinton 2009) and Tiny-Image Net.2 ... 2https://tiny-imagenet.herokuapp.com/ |
| Dataset Splits | Yes | Throughout this section, we train Pre Act Res Net-18 (He et al. 2016b) on CIFAR-10 (Krizhevsky and Hinton 2009) using standard AT (Madry et al. 2018), following the training details of Rice, Wong, and Kolter (2020). ... We report the fully trained model s accuracy and the result of the checkpoint with the best PGD accuracy (of 10 iterations), where each checkpoint is saved per epoch. ... For other training setups, we mainly follow the hyperparameters suggested by the previous studies (Pang et al. 2021; Rice, Wong, and Kolter 2020). ... In particular, our method obtained 41.2% robust accuracy even in the case when only 10% of the total dataset is accessible (where AWP achieves 34.7%). We note such efficiency is worthy for practitioners, since in such cases, validation dataset for early stopping is insufficient. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., CPU/GPU models, memory) used for running the experiments. |
| Software Dependencies | No | The paper mentions various algorithms and models used (e.g., PGD, Auto Augment, TRADES, MART) and links to external code for some, but does not provide specific version numbers for software dependencies like deep learning frameworks (e.g., TensorFlow, PyTorch) or other libraries. |
| Experiment Setup | Yes | Experimental Setups Training details. We use Pre Act-Res Net-18 (He et al. 2016b) architecture in all experiments, and additionally use Wide Res Net-34-10 (Zagoruyko and Komodakis 2016) for white-box adversarial defense on CIFAR-10. For the data augmentation, we consider Auto Augment (Cubuk et al. 2019) where random crop (with 4 pixels zero padding and horizontal flip), random horizontal flip (with 50% of probability), and Cutout (De Vries and Taylor 2017) (with half of the input width) are included. We set the regularization parameter λ = 1.0 in all cases except for applying on Wide Res Net-34-10 with TRADES and MART where we use λ = 2.0. The temperature is fixed to τ = 0.5 in all experiments. For other training setups, we mainly follow the hyperparameters suggested by the previous studies (Pang et al. 2021; Rice, Wong, and Kolter 2020). In detail, we train the network for 200 epochs3 using stochastic gradient descent with momentum 0.9, and weight decay of 0.0005. The learning rate starts at 0.1 and is dropped by a factor of 10 at 50%, and 75% of the training progress. For the inner maximization for all AT, we set the ϵ = 8/255, step size 2/255, and 10 number of steps with l constraint (see the supplementary material for the l2 constraint AT results). |