Research Reproducibility as a Survival Analysis
Authors: Edward Raff469-478
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Instead, we consider modeling the reproducibility of a paper as a survival analysis problem. We argue that this perspective represents a more accurate model of the underlying meta-science question of reproducible research, and we show how a survival analysis allows us to draw new insights that better explain prior longitudinal data. The data and code can be found at https://github.com/Edward Raff/Research-Reproducibility Survival-Analysis. The largest empirical study was done by Raff (2019), which documented features while attempting to reproduce 255 papers. This study is what we build upon for this work. Optuna (Akiba et al. 2019) was used to tune the parameters of the model resulting in a 10-fold cross-validated Concordance score of 0.80, which is a significant improvement over the linear Cox model indicating a better fit. |
| Researcher Affiliation | Collaboration | Edward Raff Booz Allen Hamilton University of Maryland, Baltimore County raff edward@bah.com raff.edward@umbc.edu |
| Pseudocode | No | No structured pseudocode or algorithm blocks (e.g., clearly labeled algorithm sections or code-like formatted procedures for the paper's own methods) were found. The paper discusses 'Pseudo Code' as a feature of other papers, but does not present its own. |
| Open Source Code | Yes | The data and code can be found at https://github.com/Edward Raff/Research-Reproducibility Survival-Analysis |
| Open Datasets | Yes | The original data used by Raff (2019) was made public but with explicit paper titles removed. We have augmented this data in order to perform this study. The data and code can be found at https://github.com/Edward Raff/Research-Reproducibility Survival-Analysis |
| Dataset Splits | Yes | Optuna (Akiba et al. 2019) was used to tune the parameters of the model resulting in a 10-fold cross-validated Concordance score of 0.80, which is a significant improvement over the linear Cox model indicating a better fit. |
| Hardware Specification | No | No specific hardware details (e.g., CPU/GPU models, memory, or cloud instance types) used for running experiments were mentioned in the paper. |
| Software Dependencies | No | No specific ancillary software details with version numbers (e.g., 'XGBoost 1.x.x', 'Optuna 2.x.x') were provided. The paper mentions 'XGBoost library' and 'Optuna (Akiba et al. 2019)' but without associated version numbers. |
| Experiment Setup | No | No specific experimental setup details, such as concrete hyperparameter values or training configurations, were provided in the main text. While 'Optuna' was used for tuning, the resulting parameters are not detailed. |