Measuring Representational Robustness of Neural Networks Through Shared Invariances
Authors: Vedant Nanda, Till Speicher, Camila Kolling, John P Dickerson, Krishna Gummadi, Adrian Weller
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Using our measure, we are able to gain insights about how shared invariances vary with changes in weight initialization, architecture, loss functions, and training dataset. Our implementation is available at: https://github. com/nvedant07/STIR. Table 1. [STIR faithfully estimates shared invariance] Here the two Res Net18s in each column are trained on CIFAR10 with different random initializations... |
| Researcher Affiliation | Collaboration | 1University of Maryland, College Park 2Max Planck Institute for Software Systems 3PUCRS 4University of Cambridge 5The Alan Turing Institute. |
| Pseudocode | No | The paper provides mathematical equations and descriptions of procedures like gradient descent but does not include any clearly labeled 'Pseudocode' or 'Algorithm' blocks. |
| Open Source Code | Yes | Our implementation is available at: https://github. com/nvedant07/STIR. |
| Open Datasets | Yes | Throughout the paper STIR is measured over CIFAR10 training samples with Sr = Linear CKA. We use Res Net18, Res Net34 (He et al., 2016), VGG16 and VGG19 (Simonyan & Zisserman, 2014) trained on CIFAR10/100 (Krizhevsky et al., 2009) |
| Dataset Splits | Yes | To simulate such a scenario, we train a Res Net18 on CIFAR10 where at each timestep, we add 5k training samples, i.e.,, at timesteps t = 0, 1, ..., T, we train the model on 5k, 10k, ..., 45k samples (we keep 5k samples for a holdout validation set). |
| Hardware Specification | No | The paper states: 'All of our experiments are performed on standard models and datasets that can fit on standard GPUs.' This description is too general and does not provide specific hardware models or detailed specifications. |
| Software Dependencies | No | The paper mentions using 'Linear CKA' and various neural network architectures and training methods (e.g., ResNet, VGG, AT, TRADES, MART) but does not list specific software libraries or frameworks with their version numbers. |
| Experiment Setup | Yes | For all purposes of adversarial training we use the ℓ2 threat model with ϵ = 1.0 (see (Madry et al., 2019) for details). Additionally TRADES and MART require another hyperparameter β (that balances adversarial robustness and clean accuracy) which we set to 1.0 for our experiments. At each timestep we train for 100 epochs. |