Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Taking Up the Gaokao Challenge: An Information Retrieval Approach
Authors: Gong Cheng, Weixi Zhu, Ziwei Wang, Jianghui Chen, Yuzhong Qu
IJCAI 2016 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our approach achieves encouraging results on real-life questions in recent history tests, significantly outperforming baseline approaches. |
| Researcher Affiliation | Academia | National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China |
| Pseudocode | Yes | Algorithm 1: Retrieving Concept Pages |
| Open Source Code | No | The paper states: 'We have made our dataset online accessible to the research community3.' with a link to the dataset, but no explicit statement or link for the source code of the methodology itself. |
| Open Datasets | Yes | We have made our dataset online accessible to the research community3. 3http://ws.nju.edu.cn/gaokao/ijcai-16/GaokaoHistory577.xml |
| Dataset Splits | No | The paper describes its dataset split into QS-A (123 questions) and QS-B (454 questions) used for evaluation. However, it does not provide explicit training, validation, or test dataset splits in terms of percentages, sample counts, or predefined partition files. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, processor types, memory amounts) used for running the experiments are provided in the paper. |
| Software Dependencies | No | The paper does not list specific software dependencies with version numbers (e.g., programming languages, libraries, frameworks, or specific solvers). |
| Experiment Setup | Yes | The parameters α = 0.8, β = 0.5, γ = 1.0 are empirically set in our experiments (Equation 1). The approach was configured to consistently use k = 6 top-ranked pages for answering every question. |