Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Statistical Regimes and Runtime Prediction
Authors: Barry Hurley, Barry O'Sullivan
IJCAI 2015 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Supported by a large-scale empirical study employing many years of industrial SAT Competition instances including repeated runs, we present statistical and empirical evidence that such a performance variation phenomenon necessitates a change in the evaluation of portfolio, runtime prediction, and automated configuration methods. |
| Researcher Affiliation | Academia | Barry Hurley and Barry O Sullivan Insight Centre for Data Analytics, University College Cork, Ireland |
| Pseudocode | No | The paper does not contain any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | No | The paper states that the dataset is available at a URL, but it does not provide an explicit statement or link for the open-source code of the methodology described in the paper. |
| Open Datasets | Yes | The dataset is available at http://ucc.insight-centre.org/bhurley/ |
| Dataset Splits | Yes | We use a standard randomised 10-fold cross-validation, where the dataset is split in 10 folds. |
| Hardware Specification | Yes | Performance data was collected on a cluster of Intel Xeon E5430 2.66Ghz processors running Cent OS 6. |
| Software Dependencies | Yes | We use Mini Sat 2.0 [E en and S orensson, 2004] as the solver... running Cent OS 6. |
| Experiment Setup | Yes | For the predictions in this section we will use the adjusted logtransformed runtime, log10(1 + runtime). ... We use a random forest with the same parameters as [Hutter et al., 2014], that is using 10 regression trees, using half of the variables as split variables at each node (perc = 0.5), a maximum of 5 data-points in leaf nodes (nmin = 5). |