Is Out-of-Distribution Detection Learnable?
Authors: Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, Feng Liu
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | To study the generalization of OOD detection, in this paper, we investigate the probably approximately correct (PAC) learning theory of OOD detection, which is proposed by researchers as an open problem. First, we find a necessary condition for the learnability of OOD detection. Then, using this condition, we prove several impossibility theorems for the learnability of OOD detection under some scenarios. Although the impossibility theorems are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios. Based on this observation, we next give several necessary and sufficient conditions to characterize the learnability of OOD detection in some practical scenarios. Lastly, we also offer theoretical supports for several representative OOD detection works based on our OOD theory. ... If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] |
| Researcher Affiliation | Academia | 1Australian Artificial Intelligence Institute, University of Technology Sydney. 2Department of Computer Sciences, University of Wisconsin-Madison. 3State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences. 4ETH Zurich, Switzerland. 5Department of Computer Science, Hong Kong Baptist University. 6School of Mathematics and Statistics, University of Melbourne. |
| Pseudocode | No | The paper describes theoretical concepts, conditions, and theorems using mathematical notation and natural language, but it does not include any explicitly labeled 'Pseudocode' or 'Algorithm' blocks. |
| Open Source Code | No | The paper explicitly marks 'N/A' for questions about experimental code, and there are no statements or links indicating the release of source code for the theoretical framework described. |
| Open Datasets | No | The paper is purely theoretical and does not conduct experiments with datasets. It mentions datasets like CIFAR-10 or MNIST as examples in discussions, but not as data used for training in its own work. The 'If you ran experiments...' section is marked N/A. |
| Dataset Splits | No | The paper is purely theoretical and does not involve experimental training. Consequently, there are no mentions of training/validation/test dataset splits. The 'If you ran experiments...' section is marked N/A. |
| Hardware Specification | No | The paper is purely theoretical and does not conduct experiments. Therefore, there is no mention of hardware specifications used for running experiments. The 'If you ran experiments...' section is marked N/A. |
| Software Dependencies | No | The paper is purely theoretical and does not involve experimental implementation. Therefore, no software dependencies with specific version numbers are mentioned. The 'If you ran experiments...' section is marked N/A. |
| Experiment Setup | No | The paper is purely theoretical and does not conduct experiments. Therefore, there are no details about an experimental setup, such as hyperparameters or system-level training settings. The 'If you ran experiments...' section is marked N/A. |