Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
Authors: Francesco Croce, Matthias Hein
ICML 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We test Auto Attack in a large-scale evaluation (Sec. 6) on over 50 classifers from 35 papers proposing robust models, including randomized defenses, from recent leading confer ences. Although using only fve restarts for each of the three white-box attacks contained in Auto Attack, in all except two cases the robust test accuracy obtained by Auto Attack is lower than the one reported in the original papers (our slightly more expensive Auto Attack+ is better in all but one case). |
| Researcher Affiliation | Academia | 1University of T ubingen, Germany. Correspondence to: F. Croce <francesco.croce@uni-tuebingen.de>. |
| Pseudocode | Yes | Algorithm 1 APGD |
| Open Source Code | Yes | Auto Attack is available at https://github.com/ fra31/auto-attack. |
| Open Datasets | Yes | We evaluate the adversarial robustness in the l1 and l2-threat models of over 50 models of 35 defenses from recent con ferences like ICML, Neur IPS, ICLR, ICCV, CVPR, using MNIST, CIFAR-10, CIFAR-100 and Image Net as datasets. |
| Dataset Splits | Yes | For each clas sifer we report the clean accuracy and robust accuracy, at the ϵ specifed in the table, on the whole test set (except for Image Net where we use 1000 points from the validation set) obtained by the individual attacks APGDCE, APGDT DLR, FABT and Square Attack, together with our ensemble Au to Attack, which counts as a success every point on which at least one of the four attacks fnds an adversarial example (worst case evaluation). |
| Hardware Specification | No | The paper does not explicitly describe the specific hardware used for running experiments (e.g., GPU/CPU models). |
| Software Dependencies | No | The paper does not provide specific version numbers for software dependencies or libraries. |
| Experiment Setup | Yes | We use 100 iterations for each run of the white-box attacks. While the runtime depends on the model, its robustness and even the framework of the target network, APGD is the fastest attack, as it requires only one forward and one backward pass per iteration. The computational budget of Auto Attack is similar to what has been used, on average, in the evaluation of the defenses considered. |