Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
A Study of the Effects of Transfer Learning on Adversarial Robustness
Authors: Pratik Vaishnavi, Kevin Eykholt, Amir Rahmati
TMLR 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this paper, we conduct an extensive analysis of the impact of transfer learning on both empirical and certified adversarial robustness. Employing supervised and self-supervised pre-training methods and fine-tuning across 12 downstream tasks representing diverse data availability scenarios, we identify the conditions conducive to training adversarially robust models through transfer learning. Our study reveals that the effectiveness of transfer learning in improving adversarial robustness is attributed to an increase in standard accuracy and not the direct transfer of robustness from the source to the target task, contrary to previous beliefs. Sections 4.2 and 4.3 present detailed experimental results in Table 4 and Table 5 respectively, showing metrics like SA, RA-WB, RA-BB, RA-CT, and ACR across various tasks. |
| Researcher Affiliation | Collaboration | Pratik Vaishnavi EMAIL Stony Brook University Kevin Eykholt EMAIL IBM Research Amir Rahmati EMAIL Stony Brook University |
| Pseudocode | No | The paper describes various methods like Adversarial Training and Consistency Regularization, and mathematical formulations for certified robustness (Equations 1 and 2), but it does not include any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | The code used to produce the findings in this paper is available at: https://github.com/Ethos-lab/transfer_learning_for_adversarial_robustness |
| Open Datasets | Yes | For pre-training (supervised and self-supervised), we use the standard Image Net dataset. For fine-tuning, we use a suite of 12 downstream datasets (Kornblith et al., 2019) often used in transfer learning literature. [...] Table 8: Statistics for all datasets used in our experiments. Dataset Image Net, Food, CIFAR-10/100, Birdsnap, SUN397, Caltech-256, Cars, Aircraft, DTD, Pets, Caltech-101, Flowers. |
| Dataset Splits | Yes | Table 8: Statistics for all datasets used in our experiments. Dataset # Train Images # Classes # Test Images Skip # Certified Images [...] Certification Using Randomized Smoothing. [...] Also following prior works, we certify about 500 test images for each dataset, by skipping every nth image in the complete test set (see Table 8 for skip factor used). |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU models, CPU types, or memory specifications used for running the experiments. It only mentions using PyTorch for training deep neural networks. |
| Software Dependencies | No | The paper mentions using 'Py Torch library (Paszke et al., 2019)', 'Robustness library (Engstrom et al., 2019)', 'Auto Attack official code', and 'code provided by Jeong et al. (Jeong & Shin, 2020)'. However, it does not specify explicit version numbers for these software dependencies, only providing citations to their respective papers or general descriptions. |
| Experiment Setup | Yes | Supervised Training: AT uses a PGD attack with ϵ = 0.5, step size = 2ϵ/3, and 3 steps. ... For CR, we use σ = 0.5, number of Gaussian noise samples m = 2, λ = 5, and η = 0.5. ... Evaluation: adversarial budget of ϵ = 0.5, ... auto PGD uses 100 steps and 5 random restarts and the Square attack uses 5000 queries and 1 random restart. ... Training: grid search over lr {0.1, 0.01, 0.05, 0.001}, batch size {256, 128, 64, 32}, and weight decay {1e 04, 1e 03, 1e 02}. ... learning rate is decayed twice by a factor of 0.1 when the performance on validation set doesn t improve for 30 epochs. For all training, we use the Stochastic Gradient Descent (SGD) optimizer. ... Certification Using Randomized Smoothing: σ = 0.5, N0 = 100, N = 100, 000, and failure probability α = 0.001. |