Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
ROOT13: Spotting Hypernyms, Co-Hyponyms and Randoms
Authors: Enrico Santus, Alessandro Lenci, Tin-Shing Chiu, Qin Lu, Chu-Ren Huang
AAAI 2016 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate it with a 10-fold cross validation on 9,600 pairs, equally distributed among the three classes and involving several Parts-Of Speech (i.e. adjectives, nouns and verbs). When all the classes are present, ROOT13 achieves an F1 score of 88.3%, against a baseline of 57.6% (vector cosine). |
| Researcher Affiliation | Academia | * The Hong Kong Polytechnic University, Hong Kong EMAIL, EMAIL, EMAIL University of Pisa, Italy EMAIL |
| Pseudocode | No | No pseudocode or clearly labeled algorithm blocks were found in the paper. |
| Open Source Code | Yes | More info: https://github.com/esantus/. |
| Open Datasets | Yes | We have used 9,600 pairs, randomly extracted from three datasets: Lenci/Benotto (Santus et al., 2014b), BLESS (Baroni and Lenci, 2011) and EVALution (Santus et al., 2015). |
| Dataset Splits | Yes | We evaluate it with a 10-fold cross validation on 9,600 pairs, equally distributed among the three classes and involving several Parts-Of Speech (i.e. adjectives, nouns and verbs). |
| Hardware Specification | No | No specific hardware details (like CPU/GPU models, memory, or specific computer specifications) used for running the experiments were provided in the paper. |
| Software Dependencies | No | The paper mentions 'Weka' for the Random Forest algorithm but does not specify its version number or any other software dependencies with version details. |
| Experiment Setup | Yes | ROOT13 uses the Random Forest algorithm implemented in Weka (Breiman, 2001), with the default settings. ... All the features are normalized in the range 0-1: ... a window-based Vector Space Model (VSM), built on a combination of uk Wa C and Wa Ckypedia corpora (around 2.7 billion words) and recording word co-occurrences within the 5 nearest content words to the left and right of each target. |