Fast Geometric Projections for Local Robustness Certification
Authors: Aymeric Fromherz, Klas Leino, Matt Fredrikson, Bryan Parno, Corina Pasareanu
ICLR 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically we find this approach to be far more precise than many approximate verification approaches, while at the same time performing multiple orders of magnitude faster than complete verifiers, and scaling to much deeper networks. In this section, we evaluate the performance of our implementation of the FGP algorithm and its variant for computing lower bounds. We performed experiments on three standard datasets: MNIST, Fashion-MNIST, and CIFAR10. We evaluated both on models trained for robustness using adversarial training (Madry et al., 2018), and on models trained for verifiability and robustness using maximum margin regularization (MMR) (Croce et al., 2019) or Re LU Stability (RS) (Xiao et al., 2019). Table 2: (a) Comparison of ℓ2 local robustness certification (FGP vs. Geo Cert vs. MIP) on 100 arbitrary test instances, including the median runtime, the certification result either robust (R), not_robust (NR), unknown (U), or a timeout (TO) and the corresponding Verified robust accuracy (VRA). |
| Researcher Affiliation | Academia | Aymeric Fromherz Carnegie Mellon University Pittsburgh, PA, USA fromherz@cmu.edu Klas Leino Carnegie Mellon University Pittsburgh, PA, USA kleino@cs.cmu.edu Matt Fredrikson Carnegie Mellon University Pittsburgh, PA, USA mfredrik@cmu.edu Bryan Parno Carnegie Mellon University Pittsburgh, PA, USA parno@cmu.edu Corina P as areanu Carnegie Mellon University and NASA Ames Moffett Field, CA, USA pcorina@cmu.edu |
| Pseudocode | No | The paper describes the Fast Geometric Projections (FGP) algorithm in detail within Section 2, but it does not include a formally structured 'Pseudocode' or 'Algorithm' block. |
| Open Source Code | Yes | An implementation of our proposed algorithm is available on Git Hub1. 1Code available at https://github.com/klasleino/fast-geometric-projections |
| Open Datasets | Yes | We performed experiments on three standard datasets: MNIST, Fashion-MNIST, and CIFAR10. |
| Dataset Splits | No | The paper states 'In each experiment, measurements are obtained by evaluating on 100 arbitrary instances from the test set' and specifies training parameters like 'trained each model for 20 epochs with a batch size of 128', but it does not explicitly provide details about training/validation/test splits (e.g., percentages or counts for each split) for its datasets. |
| Hardware Specification | Yes | All experiments were run on a 4.2GHz Intel Core i7-7700K with 32 GB of RAM, and a Tesla K80 GPU with 12 GB of RAM. |
| Software Dependencies | Yes | We used commit hash 8730aba for Geo Cert, and v0.2.1 for MIP. |
| Experiment Setup | Yes | For training, we used the ℓ2 norm, and let ϵ = 2.5 10 times the ϵ we verify with in order to have a higher fraction of verifiably-robust points. We trained each model for 20 epochs with a batch size of 128 and 50 PGD steps. We set γB = γD = 2.5, n B = 100, n D = 9, and λ = 0.5, and trained each model for 20 epochs with a batch size of 128. We weighted the RS loss by α = 2.0, using an ϵ (i.e., the distance over which the Re LU activations should remain stable) of 8/255, and weighted the PGD adversarial loss by β = 1.0, using an ϵ (i.e., the target robustness radius) of 36/255. We scheduled the ℓ1 regularization to decay from 10 2 to 10 3 over the course of training, and trained for 100 epochs with a batch size of 128. We conducted these attacks with an ℓ2 bound of ϵ = 0.25 with 1,000 PGD steps. |