Robust NAS under adversarial training: benchmark, theory, and beyond
Authors: Yongtao Wu, Fanghui Liu, Carl-Johann Simon-Gabriel, Grigorios Chrysos, Volkan Cevher
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this work, we aim to address these two challenges, making twofold contributions. First, we release a comprehensive data set that encompasses both clean accuracy and robust accuracy for a vast array of adversarially trained networks from the NASBench-201 search space on image datasets. Then, leveraging the neural tangent kernel (NTK) tool from deep learning theory, we establish a generalization theory for searching architecture in terms of clean accuracy and robust accuracy under multi-objective adversarial training. |
| Researcher Affiliation | Collaboration | Yongtao Wu LIONS, EPFL yongtao.wu@epfl.ch Fanghui Liu University of Warwick fanghui.liu@warwick.ac.uk Carl-Johann Simon-Gabriel Mirelo AI cjsg@mirelo.ai Grigorios G Chrysos University of Wisconsin-Madison chrysos@wisc.edu Volkan Cevher LIONS, EPFL volkan.cevher@epfl.ch |
| Pseudocode | Yes | Algorithm 1: Multi-objective adversarial training with stochastic gradient descent |
| Open Source Code | No | The code and the pre-trained weight will be publicly released after the paper s acceptance. |
| Open Datasets | Yes | We evaluate each network architecture on (a) CIFAR-10 (Krizhevsky et al., 2009), (b) CIFAR100 (Krizhevsky et al., 2009), and (c) Image Net-16-120 (Chrabaszcz et al., 2017). |
| Dataset Splits | No | The paper explicitly states the size of the training and test sets for CIFAR-10/100 and ImageNet-16-120 (e.g., 50,000 training images and 10,000 test images for CIFAR-10). However, it does not mention a distinct validation set or its specific split for reproducing the experiments. |
| Hardware Specification | Yes | In total, we adversarially train and evaluate 6466 3 3 58k architectures by a number of NVIDIA T4 Tensor Core GPUs. One seed for one dataset consumes approximately 34 hours on 350 GPUs. |
| Software Dependencies | No | The paper mentions general software components like "mini-batch SGD" and "one-cycle step-size schedule" and references specific algorithms (e.g., PGD attack by Madry et al., 2018). However, it does not provide specific version numbers for any libraries, frameworks (e.g., PyTorch, TensorFlow), or programming languages used, which are necessary for full software reproducibility. |
| Experiment Setup | Yes | We adopt a standard adversarial training setup via mini-batch SGD with a step-size of 0.05, momentum of 0.9, weight decay of 10 4, and batch size of 256. We train each network for 50 epochs where one-cycle step-size schedule with maximum step-size 0.1 (Smith & Topin, 2019)... Regarding the adversarial attack during training, we follow a common setting, i.e., 7 steps of projected gradient descent (PGD) with step-size 2/255 and perturbation radius ρ=8/255 (Madry et al., 2018). |