Towards Better Understanding of Training Certifiably Robust Models against Adversarial Examples
Authors: Sungyoon Lee, Woojin Lee, Jinseong Park, Jaewook Lee
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this paper, we provide empirical and theoretical analysis to answer these questions. First, we demonstrate that IBP [15] has a more favorable (smooth) loss landscape than other linear relaxationbased methods, and thus it often leads to better performance even with much looser bounds. To account for this difference, we present a unified view of IBP and linear relaxation-based methods and find that the relaxed gradient approximation (which will be defined in Definition 1) of each method plays a crucial role in its optimization behavior. Based on the analysis of the loss landscape and the optimization behavior, we propose a new certifiable training method that has a favorable loss landscape with tighter bounds. As a result, the proposed method can achieve a decent performance under a wide range of perturbations. We summarize the contributions of this study as follows: We provide empirical and theoretical analysis of the loss landscape of certifiable training methods and find that smoothness of the loss landscape is important for building certifiably robust models, in addition to the tightness of the upper bound. We find that the relaxed gradient approximation of a certifiable training method plays a major role in shaping the loss landscape, determining its optimization behavior. To verify our claims, we propose a certifiable training method with tighter bounds and a favorable loss landscape. With the two key factors, the proposed method can achieve a decent performance under a wide range of perturbations, while others with only one of the two can perform well only for a specific range of the adversarial perturbations. ... In this section, we demonstrate the proposed method satisfies two key criteria required for building certifiably robust models: 1) tightness of the upper bound on the worst-case loss, and 2) smoothness of the loss landscape. Subsequently, we evaluate the performance of the method by comparing with others certifiable training methods. Details on the experimental settings are in Appendix A. |
| Researcher Affiliation | Academia | Sungyoon Lee Korea Institute for Advanced Study (KIAS) sungyoonlee@kias.re.kr Woojin Lee Dongguk University-Seoul wj926@dgu.ac.kr Jinseong Park Seoul National Univeristy jinseong@snu.ac.kr Jaewook Lee Seoul National University jaewook@snu.ac.kr |
| Pseudocode | No | The paper does not contain any formally labeled pseudocode or algorithm blocks. It describes methods and procedures in narrative text and mathematical equations. |
| Open Source Code | Yes | Our code is available at https://github.com/sungyoon-lee/Loss Landscape Matters. |
| Open Datasets | Yes | We train our models on CIFAR-10 [23] and MNIST [1] using a small 4-layered fully-connected ReLU network and a large 9-layered fully-connected ReLU network, respectively. ... Table 1: Test errors (Standard / PGD / Verified error) of IBP, CROWN-IBP (β = 1), CAP, and OURS. ... Data ϵtest(l ) IBP CROWN-IBP (β = 1) CAP OURS MNIST ... CIFAR-10 (Shallow) ... CIFAR-10 (Deep) ... SVHN |
| Dataset Splits | No | The paper mentions `ϵt-scheduling` with `warm-up` and `ramp-up` phases, implying training and evaluation, but does not explicitly provide specific train/validation/test dataset split percentages or sample counts for reproduction. While it uses standard datasets like MNIST and CIFAR-10, it doesn't specify the exact splits used in its experiments. |
| Hardware Specification | No | The paper does not provide any specific hardware details such as GPU models, CPU types, or cloud computing instance specifications used for running the experiments. It only describes the software dependencies implicitly. |
| Software Dependencies | No | The paper states: "We use the standard training setup with a batch size of 128 and Adam optimizer with a learning rate of 0.001." While it mentions the optimizer and batch size, it does not specify any software versions (e.g., Python, PyTorch, TensorFlow, CUDA versions) that would be needed for reproducibility. |
| Experiment Setup | Yes | Details on the experimental settings are in Appendix A. ... We use the standard training setup with a batch size of 128 and Adam optimizer with a learning rate of 0.001. We use ϵt-scheduling with the warm-up (regular training, ϵt = 0) for the first 10 epochs and the ramp-up (0 ϵt ϵtrain) during epochs 10-130 where we linearly increase the perturbation radius ϵt at iteration t from 0 to the target perturbation ϵtrain. |