Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
Authors: Jonathan Uesato, Brendan O’Donoghue, Pushmeet Kohli, Aaron Oord
ICML 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we empirically study several broad categories of proposed defense strategies on the CIFAR-10 (Krizhevsky and Hinton, 2009) and Image Net (Deng et al., 2009) datasets. |
| Researcher Affiliation | Industry | 1Deep Mind. Correspondence to: Jonathan Uesato <juesato@google.com>. |
| Pseudocode | Yes | Algorithm 1 SPSA adversarial attack |
| Open Source Code | No | The paper does not provide an unambiguous statement or a direct link to the source code for the methodology described in this paper. While 'cleverhans v2.1.0' is cited in the references and one of the authors is listed as an author of cleverhans, there is no explicit statement indicating that the code for *this paper's specific methods* is being released or is available. |
| Open Datasets | Yes | CIFAR-10 (Krizhevsky and Hinton, 2009) and Image Net (Deng et al., 2009) datasets. |
| Dataset Splits | No | The paper mentions evaluating on 'CIFAR-10 test set' and 'Image Net' and states that adversarial training achieves '47% accuracy on CIFAR-10 against a gradient-based adversary'. It refers to a 'held-out test set' but does not specify explicit training/validation splits (e.g., percentages or counts), nor does it cite specific predefined splits for training or validation portions beyond implying the use of standard test sets. |
| Hardware Specification | No | The paper mentions 'efficient GPU implementation' in the context of SPSA but does not provide any specific details about the hardware used for experiments, such as GPU models, CPU types, or memory specifications. |
| Software Dependencies | Yes | The paper lists 'cleverhans v2.1.0' in its bibliography, and one of the authors of this paper (Jonathan Uesato) is also an author of 'cleverhans v2.1.0: an adversarial machine learning library'. This indicates the use of this specific versioned library. |
| Experiment Setup | Yes | The paper provides specific values for experimental parameters, such as 'Perturbation sizes ϵ are relative to images with pixel intensities in [0, 255]', 'JPEG quality 75', and 'ϵdefend = 16'. It also mentions parameters for the SPSA algorithm: 'perturbation size δ, step size α > 0, batch size n'. |