Projection Regret: Reducing Background Bias for Novelty Detection via Diffusion Models
Authors: Sungik Choi, Hankook Lee, Honglak Lee, Moontae Lee
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 4 Experiment In this section, we evaluate the out-of-distribution (OOD) detection performance of our Projection Regret. We first describe our experimental setups, including benchmarks, baselines, and implementation details of Projection Regret (Section 4.1). We then present our main results on the benchmarks (Section 4.2) and finally show ablation experiments of our design choices (Section 4.3). |
| Researcher Affiliation | Collaboration | Sungik Choi1 Hankook Lee1 Honglak Lee1 Moontae Lee1,2 1LG AI Research 2University of Illinois Chicago |
| Pseudocode | Yes | Algorithm 1 Projection Regret with Consistency Models (Py Torch-like Pseudo-code) |
| Open Source Code | No | The paper mentions using another author's code ('We use the author s code in https://github.com/ahsan Mah/msma.') but does not provide a statement or link for the source code of their own methodology. |
| Open Datasets | Yes | We use CIFAR-10/100 [30] and SVHN [31] for ID datasets, and CIFAR10/100, SVHN, LSUN [36], Image Net [37], Textures [38], and Interpolated CIFAR-10 [19] for OOD datasets. |
| Dataset Splits | No | The paper does not explicitly provide specific percentages, sample counts, or a detailed methodology for training/validation/test dataset splits. It mentions using datasets for ID vs OOD tasks, which implies a split but not a formal one for reproducibility. |
| Hardware Specification | Yes | While training the base EDM diffusion model and the consistency model, we use 8 A100 GPUs. During the out-of-distribution detection, we use 1 A100 GPU. |
| Software Dependencies | No | The paper states 'We implement the code on the Py Torch [47] framework.' but does not specify its version or any other software dependencies with version numbers. |
| Experiment Setup | Yes | For the ensemble size hyperparameter nα and nβ, we set nα = 40, nβ = 10 for CIFAR-10/SVHN experiments and nα = 100, nβ = 5 for CIFAR-100 experiment. As mentioned in the main paper, we set our base hyperparameter that shows the best performance against the rotated in-distribution dataset and set the ensemble configurations around it. We use 8 hyperparameter configurations for the CIFAR-10 and CIFAR-100 datasets and 4 hyperparameter configurations for the SVHN dataset. We train the baseline EDM [29] model for 390k iterations for all datasets. |