Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Window-Based Distribution Shift Detection for Deep Neural Networks
Authors: Guy Bar-Shalom, Yonatan Geifman, Ran El-Yaniv
NeurIPS 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In a comprehensive empirical study, we compared our CBD algorithm with the best-performing baselines, including the KS approach of [37]. |
| Researcher Affiliation | Collaboration | Guy Bar-Shalom Department of Computer Science Technion Israel Institute of Technology EMAIL Yonatan Geifman Deci.AI EMAIL Ran El-Yaniv Department of Computer Science Technion Israel Institute of Technology, Deci.AI EMAIL |
| Pseudocode | Yes | Algorithm 1: Selection with guaranteed coverage (SGC) |
| Open Source Code | Yes | Code is available at https://github.com/Bar SGuy/Window-Based-Distribution-Shift-Detection. |
| Open Datasets | Yes | Our experiments are conducted on the Image Net dataset [7], using its validation dataset as proxies for the source distribution, P. |
| Dataset Splits | Yes | To train our detectors, we randomly split the Image Net validation data (50,000) into two sets, a detection-training or source set, which is used to fit the detectors3 (49,000) and a validation set (1,000) for applying the shift. |
| Hardware Specification | No | The paper mentions conducting experiments and utilizing specific models like Res Net50, Mobile Net V3-S, and Vi T-T, but it does not provide any specific details about the hardware (e.g., GPU model, CPU type, memory) used to run these experiments. |
| Software Dependencies | No | The paper mentions 'All experiments were conducted using Py Torch' and refers to 'Sci Py s stats.ttest_1samp implementation [44]', as well as models being 'publicly available in timm s repository [45]', but it does not provide specific version numbers for these software components. |
| Experiment Setup | Yes | We evaluated the models using various window sizes, |Wk| {10, 20, 50, 100, 200, 500, 1000}. Our coverage-based detection algorithm applies SGC to Ctarget target coverages uniformly spread between the interval [0.1, 1], excluding the coverage of 1. We set Ctarget = 10, δ = 0.01, and κf(x) = 1 Entropy(x) for all our experiments |