ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms
Authors: William Yang, Byron Zhang, Olga Russakovsky
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through comprehensive experiments, we show that OOD detectors are more sensitive to covariate shift than to semantic shift, and the benefits of recent OOD detection algorithms on semantic shift detection is minimal. Our dataset and analyses provide important insights for guiding the design of future OOD detectors. |
| Researcher Affiliation | Academia | William Yang , Byron Zhang , Olga Russakovsky Department of Computer Science, Princeton University, Princeton, NJ, USA {williamyang,zishuoz,olgarus}@cs.princeton.edu |
| Pseudocode | No | The paper does not contain any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | Yes | Code and data is at https://github.com/princetonvisualai/imagenetood |
| Open Datasets | Yes | Common literature in OOD detection separates distribution shifts into semantic (label-altering) and covariate (label-preserving) shifts (Hsu et al., 2020; Tian et al., 2021; Yang et al., 2021b). Understanding detection behavior for either type of shift requires proper evaluation datasets that decouple semantic shifts from covariate shifts (Yang et al., 2021a). Many OOD detection datasets (Wang et al., 2018; Hendrycks et al., 2021; Galil et al., 2023) set Image Net-1K (Russakovsky et al., 2015) as in-distribution (ID) and subsets of Image Net-21K (Deng et al., 2009) as out-of-distribution (OOD). |
| Dataset Splits | No | The paper states that models were "trained on Image Net-1K" and mentions "Image Net-OOD" and other datasets for evaluation, but it does not specify the train/validation/test splits used for these datasets (e.g., percentages, sample counts, or citations to specific splits) to enable reproduction of the partitioning. |
| Hardware Specification | No | The paper describes the model architectures used (e.g., ResNet-50, DenseNet-121, Wide ResNet-50) but does not provide specific details about the hardware (e.g., GPU models, CPU types, memory) used for running the experiments. |
| Software Dependencies | No | The paper mentions using models "from the torchvision library" but does not specify software dependencies with version numbers (e.g., Python version, PyTorch version, CUDA version). |
| Experiment Setup | No | Hyperparameters are selected based on ablation studies on Image Net-1K done by original authors. |