Reliable Robustness Evaluation via Automatically Constructed Attack Ensembles
Authors: Shengcai Liu, Fu Peng, Ke Tang
AAAI 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct a large-scale evaluation to assess whether Auto AE can reliably evaluate adversarial robustness. Specifically, we use Auto AE to construct two AEs with a CIFAR10 l (ϵ=8/255) and a CIFAR-10 l2(ϵ=0.5) adversarial training model, and then apply them to 45 top defense models on the Robust Bench leaderboard (Croce et al. 2021). |
| Researcher Affiliation | Academia | Shengcai Liu1,2, Fu Peng2, Ke Tang1,2* 1Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology, Shenzhen 518055, China 2Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China liusc3@sustech.edu.cn, pengf2022@mail.sustech.edu.cn, tangk3@sustech.edu.cn |
| Pseudocode | Yes | Algorithm 1: Auto AE |
| Open Source Code | Yes | Code is available at https://github.com/Leeger PENG/Auto AE. |
| Open Datasets | Yes | Finally, we randomly select 5,000 instances from the training set of CIFAR-10 as the annotated training set and use two CIFAR-10 adversarial training models from Robustness library (Engstrom et al. 2019) to construct the AE for l and l2 attacks, respectively. |
| Dataset Splits | Yes | Finally, we randomly select 5,000 instances from the training set of CIFAR-10 as the annotated training set... Following the guidelines of Robust Bench, we use its integrated interface to test our AEs, where for CIFAR-10 and CIRAR-100 all the 10,000 test instances are used, while for Image Net a fixed set of 5,000 test instances are used. |
| Hardware Specification | No | The paper mentions 'GPU hours' in relation to computational costs ('GPU hours consumed by Auto AE to collect the candidate attacks performance data') but does not specify any particular GPU models, CPU models, or other detailed hardware specifications used for running experiments. |
| Software Dependencies | No | The paper states using 'implementations of APGD attack and FAB attack from the code repository of Auto Attack', 'implementations from the repository of CAA' for MT, CW, and DDN attacks, and 'Robustness library'. However, specific version numbers for these libraries or any underlying software frameworks (e.g., Python, PyTorch) are not provided. |
| Experiment Setup | Yes | We allow each attack with only one restart and set the AE s maximum total iteration steps to 1000, forcing Auto AE to construct AEs with low computational complexity (see Figure 1). Besides, we discretize the range of iteration steps of these attacks into 8 uniform-spacing values to reduce the computational costs of Auto AE. Finally... Note the parameter ϵ of the attacks in our AEs are always set in line with the defense being attacked. |