How Long Will It Take? Accurate Prediction of Ontology Reasoning Performance
Authors: Yong-Bin Kang, Jeff Z. Pan, Shonali Krishnaswamy, Wudhichart Sawangphol, Yuan-Fang Li
AAAI 2014 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our large-scale experiments on 6 state-of-the-art OWL 2 DL reasoners and more than 450 significantly diverse ontologies demonstrate that the prediction models achieve high accuracy, good generalizability and statistical significance. |
| Researcher Affiliation | Academia | Yong-Bin Kang Monash University, Australia yongbin.kang@monash.edu Jeff Z. Pan University of Aberdeen, UK jeff.z.pan@abdn.ac.uk Shonali Krishnaswamy Inst. for Infocomm Research, Singapore spkrishna@i2r.a-star.edu.sg Wudhichart Sawangphol Monash University, Australia wudhichart.sawangphol@monash.edu Yuan-Fang Li Monash University, Australia yuanfang.li@monash.edu |
| Pseudocode | Yes | Algorithm 1: Regression-based performance hotspot identification. |
| Open Source Code | No | The paper states that "The ontologies and the prediction models are available at http://bit.ly/1hSTy87", but it does not explicitly mention that the source code for the methodology is provided. |
| Open Datasets | Yes | 451 real-world, public-domain ontologies are collected, some of which from the Tones Ontology Repository and the Bio Ontology repository. ... http://owl.cs.manchester.ac.uk/repository/, http://www.bioontology.org/ |
| Dataset Splits | Yes | Lastly, the dataset of each reasoner is divided up into a training set and a test set in a 80/20 split. The training set is used for training the regression model (with 10-fold cross-validation)... Stratified sampling is performed with data points divided into 5 equal percentile groups on the response variable (reasoning time). |
| Hardware Specification | Yes | All experiments are conducted on a high-performance server running OS Linux 2.6.18 and Java 1.6 on an Intel Xeon X7560 CPU at 2.27GHz. A maximum of 32GB memory is allocated to each of the 6 reasoners to accommodate potential memory leak in reasoners from repeated invocations. |
| Software Dependencies | Yes | 6 state-of-the-art OWL 2 DL reasoners are selected for the experiment: Fa CT++ (version 1.5.3), Hermi T (version 1.3.6), JFact (version 0.9), MORe (version 0.1.6, with Hermi T as the underlying OWL 2 DL reasoner), Pellet (version 2.2.0) and Tr OWL (version 0.8). |
| Experiment Setup | Yes | Standard 10-fold cross-validation is performed to ensure the generalizalibity of the model. ... Stratified sampling is performed with data points divided into 5 equal percentile groups on the response variable (reasoning time). ... A maximum of 32GB memory is allocated to each of the 6 reasoners... We also apply a 20,000-second timeout... In our experiments we set the ratio threshold to 10% of the number of logical axioms in the ontology and l to 6. ... k is set to 1,000 in our experiments. |