Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Inference on Syntactic and Semantic Structures for Machine Comprehension
Authors: Chenrui Li, Yuanbin Wu, Man Lan
AAAI 2018 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments on the MCTest dataset demonstrate that the proposed models are highly competitive with state-of-the-art machine comprehension systems. |
| Researcher Affiliation | Academia | Chenrui Li,1 Yuanbin Wu,1, 2 Man Lan1, 2 1School of Computer Science and Software Engineering, East China Normal University 2Shanghai Key Laboratory of Multidimensional Information Processing {lcrr2011}@163.com, EMAIL |
| Pseudocode | Yes | Algorithm 1 Message passing for the tree model ... Algorithm 2 Hidden Variable Perceptron |
| Open Source Code | No | The paper does not provide any explicit statement or link regarding the availability of its source code. |
| Open Datasets | Yes | The MCTest dataset (Richardson, Burges, and Renshaw 2013) contains 660 stories written for elementary grade school level students. |
| Dataset Splits | Yes | We use the of๏ฌcial training, development and testing splitting. |
| Hardware Specification | Yes | The testing time is about 0.5s per question on a single CPU with 8G RAM. |
| Software Dependencies | No | The paper mentions 'Stanford Core NLP for coreference resolution', 'Stanford parser for dependency parsing', 'SEMAFOR frame-semantic parser', and 'word2vec' but does not specify version numbers for any of these software dependencies. |
| Experiment Setup | Yes | We tune the algorithm parameter on the development set, and set C = 0.001, M = 10. |