Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Robustness May Be at Odds with Accuracy
Authors: Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, Aleksander Madry
ICLR 2019 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirical examination In Section 2.1, we showed that the trade-off between standard accuracy and robustness might be inevitable. To examine how representative our theoretical model is of real-world datasets, we also experimentally investigate this issue on MNIST (Le Cun et al., 1998) as it is amenable to linear classifiers. |
| Researcher Affiliation | Academia | Dimitris Tsipras , Shibani Santurkar , Logan Engstrom , Alexander Turner, Aleksander M adry Massachusetts Institute of Technology EMAIL |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | MNIST: We use the simple convolution architecture from the Tensor Flow tutorial (TFM, 2017) 3. CIFAR-10: We consider a standard Res Net model (He et al., 2015a). It has 4 groups of residual layers with filter sizes (16, 16, 32, 64) and 5 residual units each 4. Footnote 3: https://github.com/Madry Lab/mnist_challenge/ Footnote 4: https://github.com/Madry Lab/cifar10_challenge/ |
| Open Datasets | Yes | We perform our experimental analysis on the MNIST (Le Cun et al., 2010), CIFAR-10 (Krizhevsky & Hinton, 2009) and (restricted) Image Net (Deng et al., 2009) datasets. |
| Dataset Splits | No | The paper presents results for "Standard accuracy (train)" and "Standard accuracy (test)" in figures and tables, but does not explicitly specify a validation set split or how it was used. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used, such as GPU models, CPU types, or memory specifications. It only mentions model architectures like ResNet. |
| Software Dependencies | No | The paper mentions using a 'Tensor Flow tutorial' and 'tensorpack repository' for model architectures but does not specify software dependencies with version numbers (e.g., TensorFlow version, Python version, specific libraries with their versions). |
| Experiment Setup | Yes | Table 2: Value of ε used for adversarial training/evaluation of each dataset and ℓp-norm. A.3 ADVERSARIAL TRAINING: We perform adversarial training to train robust classifiers following Madry et al. (2017). Specifically, we train against a projected gradient descent (PGD) adversary, starting from a random initial perturbation of the training data. For Binary MNIST: We use the cross-entropy loss and perform 100 epochs of gradient descent in training. |