Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP
Authors: Sepideh Esmaeilpour, Bing Liu, Eric Robertson, Lei Shu6568-6576
AAAI 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results on 5 benchmark datasets for OOD detection demonstrate that ZOC outperforms the baselines by a large margin. |
| Researcher Affiliation | Collaboration | 1 University of Illinois at Chicago 2 PAR Government 3 Amazon AWS AI {sesmae2, liub}@uic.edu, eric robertson@partech.com, shulindt@gmail.com |
| Pseudocode | Yes | Algorithm 1: Zero-shot Open-Set Detection |
| Open Source Code | Yes | The experimental results9 are summarized in Table 1. We use AUROC (Area Under the ROC curve) as the evaluation measure as it is the most commonly used measure for OOD detection. ZOC outperforms all baselines by a large margin. 9https://github.com/sesmae/ZOC |
| Open Datasets | Yes | The training data for fine-tuning is the training split of MS-COCO (2017 release) (Lin et al. 2014)2 which is a commonly used dataset for image captioning. We evaluate the performance of our proposed method ZOC on splits of CIFAR10, CIFAR100, CIFAR+10, CIFAR+50, and Tiny Imagenet. |
| Dataset Splits | Yes | We used MS-COCO validation dataset to choose the k value. We empirically found that the meaningful candidate unseen labels are present at top 35 level of the annotations. For CIFAR10 (Krizhevsky, Hinton et al. 2009)3 6 classes are used as in-distribution (or seen) classes. The 4 remaining classes are used as OOD (unseen) classes. The reported score is averaged over 5 splits (Openness = 13.39%). |
| Hardware Specification | No | The paper describes the architecture of the models used (CLIPimage, CLIPtext, Decodertext) and their configurations (e.g., number of layers, hidden sizes). However, it does not specify any hardware details such as GPU models, CPU types, or memory used for running the experiments. |
| Software Dependencies | No | The paper mentions key software components such as CLIP, BERT, and Adam optimizer, and states that they used the BERT large model from huggingface. However, it does not provide specific version numbers for any of these software dependencies, which would be necessary for full reproducibility. |
| Experiment Setup | Yes | We train Decodertext using Adam optimizer (Kingma and Ba 2017) with a constant learning rate of 10 5 for 25 epochs. Batch size is 128. |