Detecting Out-of-Distribution Examples with Gram Matrices
Authors: Chandramouli Shama Sastry, Sageev Oore
ICML 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically demonstrate applicability across a variety of architectures and vision datasets and, for the important and surprisingly hard task of detecting far out-ofdistribution examples, it generally performs better than or equal to state-of-the-art OOD detection methods (including those that do assume access to OOD examples). In this section, we demonstrate the effectiveness of the proposed metric using competitive deep convolutional neural network architectures such as Dense Net and Res Net on various computer vision benchmark datasets such as: CIFAR-10, CIFAR-100, SVHN, Tiny Image Net, LSUN and i SUN. |
| Researcher Affiliation | Collaboration | 1Dalhousie University/ Vector Institute. Correspondence to: Chandramouli Shama Sastry <chandramouli.sastry@gmail.com>. Acknowledgements: We thank the Canadian Institute for Advanced Research (CIFAR) and MITACS for their support. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute www. vectorinstitute.ai/#partners. |
| Pseudocode | Yes | Algorithm 1 Compute the minimum and maximum values of feature co-occurrences for each class, layer and order |
| Open Source Code | Yes | 1The code is open-sourced at https://github.com/VectorInstitute/gram-ood-detection |
| Open Datasets | Yes | For fair comparison and to aid reproducibility, we use the pretrained Res Net (He et al., 2016) and Dense Net (Huang et al., 2017) models open-sourced by Lee et al. (2018b), i.e. Res Net34 and Dense Net3 models trained on CIFAR-10, CIFAR-100 and SVHN datasets. For each of these models, we considered the corresponding test partitions as the indistribution (positive) examples. For CIFAR-10 and CIFAR100, we considered the out-of-distribution datasets used by Lee et al. (2018b): Tiny Imagenet, LSUN and SVHN. Additionally, we also considered the i SUN dataset. |
| Dataset Splits | Yes | Va The set of all validation examples. 10% of the examples not used in training are randomly chosen as validation examples. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory specifications) used for running the experiments. It mentions using pre-trained models but not the hardware for their own experimental setup. |
| Software Dependencies | No | The paper does not explicitly state specific software dependencies with version numbers (e.g., Python version, PyTorch/TensorFlow version, specific library versions). While code is open-sourced, this detail is not provided in the paper text. |
| Experiment Setup | No | The paper states: "Starting with a pre-trained network, we compute these bounds over only the training set, and then use them at test time to effectively discriminate between in-distribution samples and out-of-distribution samples." It also mentions: "a threshold, τ, for discriminating between out-of-distribution data and in-distribution data is computed as the 95th percentile of the total deviations of test data". However, it does not specify hyperparameter values such as learning rates, batch sizes, optimizers, or number of epochs for training the models or for the Gram matrix calculations themselves. It refers to pre-trained models from other works without detailing their training setup in this paper. |