Provable Robustness against Wasserstein Distribution Shifts via Input Randomization
Authors: Aounon Kumar, Alexander Levine, Tom Goldstein, Soheil Feizi
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We apply our technique to certify robustness against natural (non-adversarial) transformations of images such as color shifts, hue shifts, and changes in brightness and saturation. We obtain strong performance guarantees for the robust model under clearly visible shifts in the input images. Our experiments establish the non-vacuousness of our certificates by showing that the certified lower bound on a robust model s accuracy is higher than the empirical accuracy of an undefended model under a distribution shift. We also show provable distributional robustness against adversarial attacks. |
| Researcher Affiliation | Academia | Aounon Kumar University of Maryland College Park, MD aounon@umd.edu Alexander Levine University of Maryland College Park, MD alevine0@cs.umd.edu Tom Goldstein University of Maryland College Park, MD tomg@cs.umd.edu Soheil Feizi University of Maryland College Park, MD sfeizi@cs.umd.edu |
| Pseudocode | Yes | Algorithm 1 Prediction Input: Model µ, input instance x. Output: Robust prediction y. Randomize input: x S(x). Evaluate model: y = µ(x ). Return y. Algorithm 2 Certification Input: Accuracy function h, data distribution D, Wasserstein bound ϵ, integer n and α > 0. Output: Certified accuracy for bound ϵ. sum = 0. for i in 1 . . . n do Sample (x, y) D. Sample x S(x). Compute h(x , y). sum = sum + h(x , y) end for Compute 1 α confidence lower-bound h of E(x,y) D[ h(x, y)] using sum and n. Return h ψ(ϵ). |
| Open Source Code | No | The paper mentions using existing codebases (e.g., from Salman et al. (2019) and IBM ART package) but does not provide a statement or link for the open-sourcing of its own methodology's code. |
| Open Datasets | Yes | We experiment on two image classification datasets, namely CIFAR-10 (Krizhevsky et al.) and SVHN (Netzer et al., 2011) |
| Dataset Splits | Yes | To avoid this, we construct four data splits: Test set (10000 samples): The original CIFAR-10 test set. Never perturbed, only used in final model evaluation. Proxy training set (20000 samples): Used for the optimization of the proxy classifier model parameters θ in Equation 13 and discarded afterward. Training set (20000 samples): Perturbed using one round of the the standard 20 steps of the inner optimization of Equation 13, while keeping θ fixed. Validation set (10000 samples): Perturbed using the same method as the Training set. |
| Hardware Specification | Yes | We train the Res Net-110 models for 90 epochs which takes a few hours on a single NVIDIA Ge Force RTX 2080 Ti GPU and the Res Net-20 models for 40 epochs which takes around twenty minute on the same GPU. |
| Software Dependencies | No | The paper mentions software like the 'IBM ART package' and uses models like 'Res Net-110', but it does not provide specific version numbers for these or other software dependencies. |
| Experiment Setup | Yes | We train our models with varying levels of noise in the transformation space and evaluate their certified performance using smoothing distributions of different standard deviations. For CIFAR-10, we use Res Net-110 models trained under noise from Cohen et al. (2019). Attack hyperparameters are taken from Salman et al. (2019): We use 20 attack steps, a step size of e/10, and use 128 noise instances when computing gradients. Step size τ was set as 0.1 times the total ℓ2 ϵ bound. Training stops when training accuracy reaches a threshold value of 99%. |