Self-supervised Adversarial Robustness for the Low-label, High-data Regime
Authors: Sven Gowal, Po-Sen Huang, Aaron van den Oord, Timothy Mann, Pushmeet Kohli
ICLR 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate BYORL and pseudo-labeling on CIFAR-10 and IMAGENET and demonstrate that BYORL achieves significantly higher robustness in the low-label regime (i.e., models resulting from BYORL are up to two times more accurate). Experiments on CIFAR-10 against ℓ2 and ℓ∞ norm-bounded perturbations demonstrate that BYORL achieves near state-of-the-art robustness with as little as 500 labeled examples. |
| Researcher Affiliation | Industry | Sven Gowal*, Po-Sen Huang*, Aaron van den Oord, Timothy Mann & Pushmeet Kohli DeepMind London, United Kingdom {sgowal,posenhuang}@google.com |
| Pseudocode | No | The paper refers to 'Algorithm 1 in Croce & Hein, 2020' but does not provide pseudocode for its own method within the document. |
| Open Source Code | No | The paper does not provide concrete access to source code for the methodology described. It refers to external repositories for datasets or other tools but not its own implementation. |
| Open Datasets | Yes | We evaluate the performance of BYORL on CIFAR-10 against adversarial ℓ2 and ℓ∞ norm-bounded perturbations (CIFAR-100 and IMAGENET results are in the appendix). ... CIFAR-10 contains 60K images (i.e., 50K in the train set and 10K in the test set). ... This additional data is extracted from 80M-TINYIMAGES and consists of 500K unlabeled 32x32 images. ... We use the dataset from Carmon et al. (2019) available at https://github.com/yaircarmon/semisup-adv. |
| Dataset Splits | Yes | CIFAR-10 contains 60K images (i.e., 50K in the train set and 10K in the test set). ... For linear classifiers, we perform early stopping as suggested by Rice et al. (2020) using a separate set of 1024 validation images (we do the same when training UAT-FT models). |
| Hardware Specification | Yes | We use a batch size of 512 split over 32 Google Cloud TPU v3 cores. |
| Software Dependencies | No | The paper mentions various software components and optimizers like LARS, Adam, Batch Normalization, and ReLU, and refers to external attacks like Auto Attack. However, it does not provide specific version numbers for these software dependencies, e.g., 'PyTorch 1.9' or 'TensorFlow 2.x'. |
| Experiment Setup | Yes | Architecture. We use a convolutional residual network (He et al., 2015) with 34 layers (Pre Activation Res Net-34) as our encoder e. We also use wider (from 1 to 4) Res Nets. The projector g and predictor q networks are MLPs with hidden dimension 4096 and output dimension 256. ... Outer optimization. We use the LARS optimizer (You et al., 2017) with a cosine learning rate schedule (Loshchilov & Hutter, 2017) over 1000 epochs. We set the learning rate to 2 and use a global weight decay parameter of 5e-4. For the target network, the exponential moving average parameter τ starts from 0.996 and is increased to one during training. We use a batch size of 512. ... Inner optimization. The inner minimization in Eq. 7 is implemented using K PGD steps (constrained by an ℓ2 or ℓ∞ norm-bounded ball). Unless specified otherwise, we set K to 40 and use an adaptive step size α (see Algorithm 1 in Croce & Hein, 2020). |