Empirically Measuring Concentration: Fundamental Limits on Intrinsic Robustness
Authors: Saeed Mahloujifar, Xiao Zhang, Mohammad Mahmoody, David Evans
NeurIPS 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This paper presents a method for empirically measuring and bounding the concentration of a concrete dataset which is proven to converge to the actual concentration. We use it to empirically estimate the intrinsic robustness to ℓ and ℓ2 perturbations of several image classification benchmarks. |
| Researcher Affiliation | Academia | Saeed Mahloujifar , Xiao Zhang , Mohammad Mahmoody, and David Evans University of Virginia [saeed, shawn, mohammad, evans]@virginia.edu |
| Pseudocode | Yes | for pseudocode, see Algorithm 1 in Appendix B |
| Open Source Code | Yes | Code for our experiments is available at https://github.com/xiaozhanguva/Measure-Concentration. |
| Open Datasets | Yes | We evaluate our approach on two benchmark image datasets: MNIST (Le Cun et al., 2010) and CIFAR-10 (Krizhevsky & Hinton, 2009). |
| Dataset Splits | No | Given the number of hyperrectangles, T, we obtain the resulting error region using the proposed algorithm on the training dataset, and tune T for the minimum adversarial risk on the testing dataset. |
| Hardware Specification | No | The paper does not provide any specific hardware details such as GPU models, CPU types, or memory specifications used for experiments. |
| Software Dependencies | No | The paper mentions 'scikit-learn: machine learning in Python' in its references, but does not provide specific software dependencies with version numbers used for running the experiments. |
| Experiment Setup | Yes | More specifically, our algorithm... starts by sorting all the training images in an ascending order based on the ℓ1-norm distance to the k-th nearest neighbour with k = 50, and then obtains T hyperrectangular image clusters by performing k-means clustering... where the metric is chosen as ℓ1 and the maximum iterations is set as 30. Finally, we perform a binary search over q [0, 1], where we set δbin = 0.005 as the stopping criteria... We choose α to reflect the best accuracy achieved by state-of-the-art classifiers, using α = 0.01 and ϵ {0.1, 0.2, 0.3, 0.4} for MNIST and selecting appropriate values to represent the best typical results on the other datasets (see Table 1). Given the number of hyperrectangles, T, we obtain the resulting error region using the proposed algorithm on the training dataset, and tune T for the minimum adversarial risk on the testing dataset. |