Intriguing Properties of Adversarial Training at Scale
Authors: Cihang Xie, Alan Yuille
ICLR 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this paper, we provide the first rigorous study on diagnosing elements of large-scale adversarial training on Image Net, which reveals two intriguing properties. First, we study the role of normalization. Batch Normalization (BN) is a crucial element for achieving state-of-the-art performance on many vision tasks, but we show it may prevent networks from obtaining strong robustness in adversarial training. One unexpected observation is that, for models trained with BN, simply removing clean images from training data largely boosts adversarial robustness, i.e., 18.3%. We relate this phenomenon to the hypothesis that clean images and adversarial images are drawn from two different domains. This two-domain hypothesis may explain the issue of BN when training with a mixture of clean and adversarial images, as estimating normalization statistics of this mixture distribution is challenging. Guided by this two-domain hypothesis, we show disentangling the mixture distribution for normalization, i.e., applying separate BNs to clean and adversarial images for statistics estimation, achieves much stronger robustness. Additionally, we find that enforcing BNs to behave consistently at training and testing can further enhance robustness. Second, we study the role of network capacity. We find our so-called deep networks are still shallow for the task of adversarial learning. Unlike traditional classification tasks where accuracy is only marginally improved by adding more layers to deep networks (e.g., Res Net-152), adversarial training exhibits a much stronger demand on deeper networks to achieve higher adversarial robustness. This robustness improvement can be observed substantially and consistently even by pushing the network capacity to an unprecedented scale, i.e., Res Net-638. |
| Researcher Affiliation | Academia | Cihang Xie Johns Hopkins University Alan Yuille Johns Hopkins University |
| Pseudocode | No | No pseudocode or algorithm blocks were found in the paper. |
| Open Source Code | No | We use the publicly available adversarial training pipeline1 to train all models with different strategies on Image Net. 1https://github.com/facebookresearch/ImageNet-Adversarial-Training |
| Open Datasets | Yes | In this paper, we provide the first rigorous diagnosis of different adversarial learning strategies, under a unified training and testing framework, on the large-scale Image Net dataset (Russakovsky et al., 2015). |
| Dataset Splits | No | We use the publicly available adversarial training pipeline1 to train all models with different strategies on Image Net. We select Res Net-152 (He et al., 2016) as the baseline network, and apply projected gradient descent (PGD) (Madry et al., 2018) as the adversarial attacker to generate adversarial examples during training. All models are trained for a total of 110 epochs... |
| Hardware Specification | No | The original ALP uses 50 GPUs for adversarial training, while ours uses 128 GPUs. |
| Software Dependencies | No | No specific software dependencies with version numbers (e.g., Python 3.8, PyTorch 1.9) were explicitly mentioned in the paper. |
| Experiment Setup | Yes | We select Res Net-152 (He et al., 2016) as the baseline network, and apply projected gradient descent (PGD) (Madry et al., 2018) as the adversarial attacker to generate adversarial examples during training. The hyper-parameters of the PGD attacker are: maximum perturbation of each pixel ϵ = 16, attack step size α = 1, number of attack iterations N = 30, and the targeted class is selected uniformly at random over the 1000 Image Net categories. We initialize the adversarial image by the clean counterpart with probability = 0.2, or randomly within the allowed ϵ cube with probability = 0.8. All models are trained for a total of 110 epochs, and we decrease the learning rate by 10 at the 35-th, 70-th, and 95-th epoch. The default number of training batch size is 4096. |