Self-ensemble Adversarial Training for Improved Robustness
Authors: Hongjun Wang, Yisen Wang
ICLR 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We also discuss the relationship between the ensemble of predictions from different adversarially trained models and the prediction of weight-ensembled models, as well as provide theoretical and empirical evidence that the proposed self-ensemble method provides a smoother loss landscape and better robustness than both individual models and the ensemble of predictions from different classifiers. We further analyze a subtle but fatal issue in the general settings for the self-ensemble model, which causes the deterioration of the weight-ensembled method in the late phases*. ... We thoroughly compare adversarial robustness of SEAT with other models trained by the state-of-the-art techniques against several attack methods on CIFAR-10 and CIFAR-100 datasets. Results have shown that the SEAT method itself can efficiently search in the weight space and significantly improve adversarial robustness. |
| Researcher Affiliation | Academia | Hongjun Wang1 Yisen Wang1,2 1 Key Lab. of Machine Perception (Mo E), School of Artificial Intelligence, Peking University 2 Institute for Artificial Intelligence, Peking University |
| Pseudocode | Yes | Algorithm 1 Self-Ensemble Adversarial Training (SEAT) |
| Open Source Code | Yes | Code is available at https://github.com/whj363636/Self-Ensemble-Adversarial-Training |
| Open Datasets | Yes | We mainly use Res Net18 and WRN-32-10 (Zagoruyko & Komodakis, 2016) for the experiments on CIFAR-10/CIFAR-100 and all images are normalized into [0, 1]. |
| Dataset Splits | No | The paper mentions using CIFAR-10/100 datasets and a 'testing dataset', but it does not explicitly provide details for training/validation/test splits, percentages, or sample counts, nor does it refer to predefined splits with citations for the splits themselves. |
| Hardware Specification | Yes | All the methods were realized by Pytorch 1.5, where we used a single NVIDIA Ge Force RTX 3090 GPU. |
| Software Dependencies | Yes | All the methods were realized by Pytorch 1.5, where we used a single NVIDIA Ge Force RTX 3090 GPU. |
| Experiment Setup | Yes | We train Res Net18 using SGD with 0.9 momentum for 120 epochs and the weight decay factor is set to 3.5e 3 for Res Net18 and 7e 4 for WRN-32-10. For SEAT, we use the piecewise linear learning rate schedule instead of the staircase one based on Proposition 2. The initial learning rate for Res Net18 is set to 0.01 and 0.1 for WRN-32-10 till Epoch 40 and then linearly reduced to 0.001, 0.0001 and 0.01, 0.001 at Epoch 60 and 120, respectively. The magnitude of maximum perturbation at each pixel is ε = 8/255 with step size κ = 2/255 and the PGD steps number in the inner maximization is 10. |