Out-of-Distribution Detection with Deep Nearest Neighbors
Authors: Yiyou Sun, Yifei Ming, Xiaojin Zhu, Yixuan Li
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the effectiveness of nearest-neighbor-based OOD detection on several benchmarks and establish superior performance. Under the same model trained on Image Net-1k, our method substantially reduces the false positive rate (FPR@TPR95) by 24.77% compared to a strong baseline SSD+, which uses a parametric approach Mahalanobis distance in detection. Code is available: https://github.com/ deeplearning-wisc/knn-ood. |
| Researcher Affiliation | Academia | 1Department of Computer Sciences, University of Wisconsin Madison. Correspondence to: Yiyou Sun, Yixuan Li <sunyiyou, sharonli@cs.wisc.edu>. |
| Pseudocode | Yes | Algorithm 1 OOD Detection with Deep Nearest Neighbors Input: Training dataset Din, pre-trained neural network encoder ϕ, test sample x , threshold λ For xi in the training data Din, collect feature vectors Zn = (z1, z2, ..., zn) Testing Stage: Given a test sample, we calculate feature vector z = ϕ(x )/ ϕ(x ) 2 Reorder Zn according to the increasing value of zi z 2 as Z n = (z(1), z(2), ..., z(n)) Output: OOD detection decision 1{ z z(k) 2 λ} |
| Open Source Code | Yes | Code is available: https://github.com/deeplearning-wisc/knn-ood. |
| Open Datasets | Yes | Datasets We begin with the CIFAR benchmarks that are routinely used in literature. We use the standard split with 50,000 training images and 10,000 test images. We evaluate the methods on common OOD datasets: Textures (Cimpoi et al., 2014), SVHN (Netzer et al., 2011), Places365 (Zhou et al., 2017), LSUN-C (Yu et al., 2015), and i SUN (Xu et al., 2015). |
| Dataset Splits | Yes | We use the standard split with 50,000 training images and 10,000 test images. We use k = 50 for CIFAR-10 and k = 200 for CIFAR-100, which is selected from k = {1, 10, 20, 50, 100, 200, 500, 1000, 3000, 5000} using the validation method in (Hendrycks et al., 2019). |
| Hardware Specification | No | No specific hardware models (e.g., GPU, CPU) or cloud instance types are mentioned for running the experiments. The paper only mentions using PyTorch and Faiss, and refers to external work's hardware (CSI's 8 Nvidia 2080Tis) not their own. |
| Software Dependencies | No | All of the experiments are based on Py Torch (Paszke et al., 2019). We use Faiss (Johnson et al., 2019), a library for efficient nearest neighbor search. |
| Experiment Setup | Yes | Experiment details We use Res Net-18 as the backbone for CIFAR-10. Following the original settings in Khosla et al., models with Sup Con loss are trained for 500 epochs, with the batch size of 1024. The temperature τ is 0.1. The dimension of the penultimate feature where we perform the nearest neighbor search is 512. The dimension of the projection head is 128. We use the cosine annealing learning rate (Loshchilov & Hutter, 2016) starting at 0.5. We use k = 50 for CIFAR-10 and k = 200 for CIFAR-100... We train the models using stochastic gradient descent with momentum 0.9, and weight decay 10 4. The model without contrastive learning is trained for 100 epochs. The start learning rate is 0.1 and decays by a factor of 10 at epochs 50, 75, and 90 respectively. |