Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples
Authors: Maura Pintor, Luca Demetrio, Angelo Sotgiu, Ambra Demontis, Nicholas Carlini, Battista Biggio, Fabio Roli
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our extensive experimental analysis, involving more than 15 models in 3 distinct application domains, shows that our indicators of failure can be used to debug and improve current adversarial robustness evaluations, thereby providing a first concrete step towards automatizing and systematizing them. |
| Researcher Affiliation | Collaboration | Maura Pintor University of Cagliari, Italy Pluribus One, Italy maura.pintor@unica.it Luca Demetrio University of Genoa, Italy Pluribus One, Italy luca.demetrio@unige.it Angelo Sotgiu University of Cagliari, Italy angelo.sotgiu@unica.it Ambra Demontis University of Cagliari, Italy ambra.demontis@unica.it Nicholas Carlini Google nicholas@carlini.com Battista Biggio University of Cagliari, CINI, Italy Pluribus One, Italy battista.biggio@unica.it Fabio Roli University of Genoa, CINI, Italy Pluribus One, Italy fabio.roli@unige.it |
| Pseudocode | Yes | Algorithm 1: Generalized gradient-based attack for optimizing adversarial examples. |
| Open Source Code | Yes | Our open-source code is available at: https://github.com/pralab/Indicators Of Attack Failure. |
| Open Datasets | Yes | All the robustness evaluations are performed on 100 samples from the test dataset of the considered model, and for each attack we evaluate the robust accuracy with ϵ = 8/255 for CIFAR models, and ϵ = 0.5 for the MNIST ones. |
| Dataset Splits | No | The paper mentions evaluating on a 'test dataset' but does not specify the explicit training, validation, and test splits (e.g., percentages or counts for all splits) needed for full reproduction of data partitioning. |
| Hardware Specification | Yes | We run our attacks on an Intel Xeon CPU E5-2670 v3, with 48 cores, 128 GB of RAM, equipped with a Nvidia Quadro M6000 with 24 GB of memory |
| Software Dependencies | No | The paper mentions leveraging the 'Sec ML library [32]' but does not provide a specific version number for this library or any other software components used in the experiments. |
| Experiment Setup | Yes | For I1, I2, and I6 we set N = 10, and for I4 we set k = 10. For I2, we set the number of sampled neighboors s = 100, and the radius of the ℓ2 ball r = 10 3, to match the step size α of the evaluations. The thresholds τ of I2 and µ of I4 are set to 10% and 1% respectively (details about their calibration in A.2). All the robustness evaluations are performed on 100 samples from the test dataset of the considered model, and for each attack we evaluate the robust accuracy with ϵ = 8/255 for CIFAR models, and ϵ = 0.5 for the MNIST ones. The step size α is set to match the original evaluations (as detailed in A.1). |