A FRAMEWORK FOR ROBUSTNESS CERTIFICATION OF SMOOTHED CLASSIFIERS USING F-DIVERGENCES
Authors: Krishnamurthy (Dj) Dvijotham, Jamie Hayes, Borja Balle, Zico Kolter, Chongli Qin, Andras Gyorgy, Kai Xiao, Sven Gowal, Pushmeet Kohli
ICLR 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate our framework experimentally on image and classification tasks, obtaining robustness certificates that improve upon other black-box methods either in terms of certificate tightness or computation time on robustness to ℓ0, ℓ1 or ℓ2 perturbations on MNIST, CIFAR-10 and Image Net. ℓ2 perturbations result from worst-case realizations of white noise that is common in many image, speech and video processing. ℓ0 perturbations can model missing data (missing pixels in an image, or samples in a time-domain audio signal) while ℓ1 perturbations can be used to model convex combinations of discrete perturbations in text classification (Jia et al., 2019). We also obtain the first, to the best of our knowledge, certifiably robust model for an audio classification task, Librispeech (Panayotov et al., 2015), with variable-length inputs. |
| Researcher Affiliation | Collaboration | Deep Mind, London, UK University College London, UK a Carnegie Mellon University, USA b Massachusetts Institute of Technology, USA c Bosch Center for AI, USA {dvij, bballe, chongliqin, agyorgy, sgowal, pushmeet}@google.com {zkolter}@cs.cmu.edu {j.hayes}@ucl.ac.uk {kaix}@mit.edu |
| Pseudocode | Yes | Algorithm 1 Full information certification (see appendix A.9 for details of subroutines) Inputs: Query access to specification φ : X [a, b], sampling access to reference distribution ρ, divergences fi and bounds ϵi, sample sizes N, N, confidence level ζ. 1: κ , λ ESTIMATEOPT(ρ, φ, N, {fi}M i=1, {ϵi}M i=1). 2: Eub UPPERCONFIDENCEBOUND(ρ, φ, N, {fi}M i=1, {ϵi}M i=1, a, b, λ , κ , ζ). 3: If κ PM i=1 λ i ϵi Eub 0 return CERTIFIED else return NOT CERTIFIED. |
| Open Source Code | No | The paper mentions using a model "released in the Github code accompanying the paper of Lee et al. (2019)", but does not provide a statement or link for the code developed for *this* paper. |
| Open Datasets | Yes | We evaluate our framework experimentally on image and classification tasks, obtaining robustness certificates that improve upon other black-box methods either in terms of certificate tightness or computation time on robustness to ℓ0, ℓ1 or ℓ2 perturbations on MNIST, CIFAR-10 and Image Net. ℓ2 perturbations result from worst-case realizations of white noise that is common in many image, speech and video processing. ℓ0 perturbations can model missing data (missing pixels in an image, or samples in a time-domain audio signal) while ℓ1 perturbations can be used to model convex combinations of discrete perturbations in text classification (Jia et al., 2019). We also obtain the first, to the best of our knowledge, certifiably robust model for an audio classification task, Librispeech (Panayotov et al., 2015), with variable-length inputs. |
| Dataset Splits | No | The paper mentions evaluating on a validation set (e.g., "Image Net validation set images" in A.11), but does not provide specific details on the train/validation/test split percentages or sample counts for the overall datasets used. |
| Hardware Specification | No | The paper does not explicitly describe the specific hardware (e.g., GPU models, CPU types, memory amounts) used for running its experiments. |
| Software Dependencies | No | The paper mentions "CVXPY" and types of models like "Res Net-152" but does not provide specific version numbers for any software dependencies. |
| Experiment Setup | Yes | MNIST hyperparameters: We trained a standard three layer CNN Re LU classifier for 50,000 steps with a batch size of 128 and a learning rate of 0.001. The smoothing value during training was set to 1.0. For certification we use N = 1K, N = 10M, ζ = .99, and sweep over a range of smoothing values between 0.5 and 1.5 and report the best certificate found. CIFAR-10 hyperparameters: We trained a Wide Res Net classifier for 50,000 training steps with a batch size of 32 and a learning rate of 0.001. The smoothing value during training was set to 0.2. For certification we use N = 1K, N = 1M, ζ = .99, and sweep over a range of smoothing values between 0.1 and 0.5 and report the best certificate found. Image Net hyperparameters: We trained a Res Net-152 classifier for 1 million training steps with a batch size of 16 and an initial learning rate of 0.1 that is decayed by a factor of ten every 25,000 steps. The smoothing value during training was set to 0.1. For certification we use N = 1K, N = 100K, ζ = .99, and sweep over a range of smoothing values between 0.05 and 0.25 and report the best certificate found. |