Connecting Certified and Adversarial Training
Authors: Yuhao Mao, Mark Müller, Marc Fischer, Martin Vechev
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, TAPS achieves a new state-of-the-art in many settings, e.g., reaching a certified accuracy of 22% on TINYIMAGENET for ℓ -perturbations with radius ϵ = 1/255. We make our implementation and networks public at github.com/eth-sri/taps. In this section, we evaluate TAPS empirically, first, comparing it to a range of state-of-the-art certified training methods, before conducting an extensive ablation study validating our design choices. |
| Researcher Affiliation | Academia | Yuhao Mao, Mark Niklas Müller, Marc Fischer, Martin Vechev Department of Computer Science ETH Zurich, Switzerland {yuhao.mao, mark.mueller, marc.fischer, martin.vechev}@inf.ethz.ch |
| Pseudocode | Yes | Algorithm 1 Train Loss Computation Input: data XB = {(xb, yb)}b, current ϵ, target ϵt, network f Output: A differentiable loss L LIBP = P b B LIBP(xb, yb, ϵ)/|B|. if ϵ < ϵt then // ϵ annealing regularisation from Shi et al. (2021) Lfast = λ (Ltightness + Lrelu) return LIBP + ϵ/ϵt Lfast LTAPS = P b B LTAPS(xb, yb, ϵ)/|B|. return LIBP LTAPS |
| Open Source Code | Yes | We make our implementation and networks public at github.com/eth-sri/taps. We publish our code, trained models, and detailed instructions on how to reproduce our results at github.com/eth-sri/taps, providing an anonymized version to the reviewers3. Additionally, we provide detailed descriptions of all hyper-parameter choices, data sets, and preprocessing steps in App. B. 3We provide the codebase with the supplementary material, including instructions on how to download our trained models. |
| Open Datasets | Yes | We use the MNIST (Le Cun et al., 2010), CIFAR-10 (Krizhevsky et al., 2009), and TINYIMAGENET (Le and Yang, 2015) datasets, all of which are freely available with no license specified. The data preprocessing mostly follows Müller et al. (2022a). |
| Dataset Splits | Yes | We follow Shi et al. (2021) in doing early stopping based on validation set performance. However, we use TAPS accuracy (see App. C) instead of IBP accuracy as a performance metric. For CIFAR-10 and TINYIMAGENET, we normalize with the dataset mean and standard deviation (after calculating perturbation size) and augment with random horizontal flips. For CIFAR-10, we apply random cropping to 32 32 after applying a 2 pixel padding at every margin. For TINYIMAGENET, we apply random cropping to 56 56 during training and center cropping during testing. |
| Hardware Specification | Yes | We train using single NVIDIA Ge Force RTX 3090 for MNIST and CIFAR-10 and single NVIDIA TITAN RTX for TINYIMAGENET. The certification is run on a single NVIDIA TITAN RTX. |
| Software Dependencies | No | The paper mentions key software components like Py Torch and MN-BAB but does not specify their version numbers, which is required for reproducibility. |
| Experiment Setup | Yes | Experimental Setup We implement TAPS in Py Torch (Paszke et al., 2019) and use MN-BAB (Ferrari et al., 2022) for certification. We conduct experiments on MNIST (Le Cun et al., 2010), CIFAR-10 (Krizhevsky et al., 2009), and TINYIMAGENET (Le and Yang, 2015) using ℓ perturbations and the CNN7 architecture (Gowal et al., 2018). For more experimental details including hyperparameters and computational costs and an extended analysis see App. B and App. C, respectively. We follow the hyperparameter choices of Shi et al. (2021) for ϵ-annealing, learning rate schedules, batch sizes, and gradient clipping (see Table 5). We set the initial learning rate to 0.0005 and decrease it by a factor of 0.2 at Decay-1 and -2. We set the gradient clipping threshold to 10. We use additional L1 regularization in some settings where we observe signs of overfitting. We report the L1 regularization and split position chosen for different settings in Table 6 and Table 8. |