Window-Based Distribution Shift Detection for Deep Neural Networks
Authors: Guy Bar-Shalom, Yonatan Geifman, Ran El-Yaniv
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In a comprehensive empirical study, we compared our CBD algorithm with the best-performing baselines, including the KS approach of [37]. |
| Researcher Affiliation | Collaboration | Guy Bar-Shalom Department of Computer Science Technion Israel Institute of Technology guy.b@cs.technion.ac.il Yonatan Geifman Deci.AI yonatan@deci.ai Ran El-Yaniv Department of Computer Science Technion Israel Institute of Technology, Deci.AI rani@cs.technion.ac.il |
| Pseudocode | Yes | Algorithm 1: Selection with guaranteed coverage (SGC) |
| Open Source Code | Yes | Code is available at https://github.com/Bar SGuy/Window-Based-Distribution-Shift-Detection. |
| Open Datasets | Yes | Our experiments are conducted on the Image Net dataset [7], using its validation dataset as proxies for the source distribution, P. |
| Dataset Splits | Yes | To train our detectors, we randomly split the Image Net validation data (50,000) into two sets, a detection-training or source set, which is used to fit the detectors3 (49,000) and a validation set (1,000) for applying the shift. |
| Hardware Specification | No | The paper mentions conducting experiments and utilizing specific models like Res Net50, Mobile Net V3-S, and Vi T-T, but it does not provide any specific details about the hardware (e.g., GPU model, CPU type, memory) used to run these experiments. |
| Software Dependencies | No | The paper mentions 'All experiments were conducted using Py Torch' and refers to 'Sci Py s stats.ttest_1samp implementation [44]', as well as models being 'publicly available in timm s repository [45]', but it does not provide specific version numbers for these software components. |
| Experiment Setup | Yes | We evaluated the models using various window sizes, |Wk| {10, 20, 50, 100, 200, 500, 1000}. Our coverage-based detection algorithm applies SGC to Ctarget target coverages uniformly spread between the interval [0.1, 1], excluding the coverage of 1. We set Ctarget = 10, δ = 0.01, and κf(x) = 1 Entropy(x) for all our experiments |