Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them
Authors: Florian Tramer
ICML 2022 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To illustrate, we revisit 14 empirical detector defenses published over the past years. For 12/14 defenses, we show that the claimed detection results imply an inefficient classifier with robustness far beyond the state-of-the-art. [...] We now survey 14 detection defenses, and consider the robust classification performance that these defenses implicitly claim (via Theorem 4). As we will see, in 12/14 cases, the defenses detection results imply a computationally inefficient classifier with far better robust accuracy than the state-of-the-art. |
| Researcher Affiliation | Collaboration | 1Google Research 2Work done while the author was at Stanford University. Correspondence to: Florian Tram er <EMAIL>. |
| Pseudocode | No | The paper describes algorithms in prose and bullet points, but not in a formally labeled 'Pseudocode' or 'Algorithm' block or figure. |
| Open Source Code | No | The paper references third-party code (e.g., 'Robustness (python library), 2019. URL https://github.com/Madry Lab/ robustness') but does not state that the code for its own methodology or analysis is open-source or provide a link to it. |
| Open Datasets | Yes | The 14 detector defenses use three datasets: MNIST, CIFAR-10 and Image Net, and consider adversarial examples under the ℓ or ℓ2 norms. |
| Dataset Splits | No | The paper mentions using adversarially-trained classifiers from other works, but it does not specify the training, validation, or test dataset splits used for reproducibility in its own analysis. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory) used for running its experiments or analysis. |
| Software Dependencies | No | The paper cites external software like 'Robustness (python library), 2019' but does not provide a list of its own specific ancillary software dependencies with version numbers needed to replicate its analysis. |
| Experiment Setup | No | The paper describes how existing detection defense claims were analyzed and contrasted with state-of-the-art robust classification, and the formula used for bounding robust risk. However, it does not provide specific hyperparameters or training configurations for a model built or trained by the authors for their analysis. |