A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks
Authors: Hadi Salman, Greg Yang, Huan Zhang, Cho-Jui Hsieh, Pengchuan Zhang
NeurIPS 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Next, we perform large-scale experiments, amounting to more than 22 CPU-years, to obtain exact solution to the convex-relaxed problem that is optimal within our framework for Re LU networks. We find the exact solution does not significantly improve upon the gap between PGD and existing relaxed verifiers for various networks trained normally or robustly on MNIST and CIFAR datasets. Our results suggest there is an inherent barrier to tight verification for the large class of methods captured by our framework. |
| Researcher Affiliation | Collaboration | Hadi Salman Microsoft Research AI hadi.salman@microsoft.com Greg Yang Microsoft Research AI gregyang@microsoft.com Huan Zhang UCLA huan@huan-zhang.com Cho-Jui Hsieh UCLA chohsieh@cs.ucla.edu Pengchuan Zhang Microsoft Research AI penzhan@microsoft.com |
| Pseudocode | No | The paper references "Algorithm 1 of Wong and Kolter [2018]" but does not contain its own structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our code and trained models are available at http://github.com/Hadisalman/robust-verify-benchmark2. |
| Open Datasets | Yes | All experiments are conducted on MNIST and/or CIFAR-10 datasets. |
| Dataset Splits | No | The paper mentions the use of 'test set' and 'training methods', but it does not provide specific details on train/validation/test dataset splits (e.g., percentages, sample counts, or explicit splitting methodology) within the main text. |
| Hardware Specification | Yes | We run experiments on a cluster with 1000 CPU-nodes. The total run time amounts to more than 22 CPU-years. |
| Software Dependencies | No | The paper mentions software components like CVXPY, ECOS, and refers to methods from other papers (e.g., Wong and Kolter [2018], Tjeng et al. [2019]), but it does not specify version numbers for these or any other ancillary software used in their own implementation or experiments. |
| Experiment Setup | Yes | Architectures. We conduct experiments on a range of Re LU-activated feedforward networks. MLP-A and MLP-B refer to multilayer perceptrons: MLP-A has 1 hidden layer with 500 neurons, and MLP-B has 2 hidden layers with 100 neurons each. CNN-SMALL, CNN-WIDE-K, and CNNDEEP-K are the Conv Net architectures used in Wong et al. [2018]. Full details are in Appendix I.1. Training Modes. We conduct experiments on networks trained with a regular cross-entropy (CE) loss function and networks trained to be robust. These networks are identified by a prefix corresponding to the method used to train them: LPD when the LP-relaxed dual formulation of Wong and Kolter [2018] is used for robust training, ADV when adversarial examples generated using PGD are used for robust training, as in Madry et al. [2017], and NOR when the network is normally trained using the CE loss function. Training details are in Appendix I.2. |