Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Progressive Comparison for Ranking Estimation
Authors: Ryusuke Takahama, Toshihiro Kamishima, Hisashi Kashima
IJCAI 2016 | Venue PDF | LLM Run Details | Input Tokens: 13,667 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 3,895 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The results of experiments using a synthetic dataset and two real datasets demonstrate that the progressive comparison method achieves high estimation accuracy with a smaller number of evaluations than the standard pairwise comparison method, and that the active learning methods further reduce the number of evaluations as compared with a random sampling method. |
| Researcher Affiliation | Academia | 1Kyoto University 2JST, ERATO, Kawarabayashi Large Graph Project 3National Institute of Advanced Industrial Science and Technology (AIST) |
| Pseudocode | No | No pseudocode or clearly labeled algorithm block found. |
| Open Source Code | No | The paper does not provide a link or explicit statement about the availability of open-source code for the described methodology. |
| Open Datasets | Yes | The dataset is available at http://goo.gl/6MS9MK. |
| Dataset Splits | No | The paper does not explicitly provide training/test/validation dataset splits. It describes initialization parameters and comparing estimated ranking lists with true ones, but no specific data partitioning. |
| Hardware Specification | No | The paper does not explicitly describe the hardware used to run its experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details with version numbers. |
| Experiment Setup | Yes | Throughout the experiments, the parameters are initialized ยตi = 1, 500 and ฯ2i = 1472 = 21, 609 by following the settings of the glicko update equations [Glickman, 1999]. |