Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
PAGOdA: Pay-As-You-Go Ontology Query Answering Using a Datalog Reasoner
Authors: Yujiao Zhou, Bernardo Cuenca Grau, Yavor Nenov, Mark Kaminski, Ian Horrocks
JAIR 2015 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our extensive evaluation shows that PAGOd A succeeds in providing scalable pay-as-you-go query answering for a wide range of OWL 2 ontologies, datasets and queries. (Abstract) and We have implemented our techniques in the PAGOd A system... and conducted an extensive evaluation using a wide range of realistic and benchmark datasets and queries. (Introduction). Section 10 is dedicated to Evaluation. |
| Researcher Affiliation | Academia | Yujiao Zhou EMAIL Bernardo Cuenca Grau EMAIL Yavor Nenov EMAIL Mark Kaminski EMAIL Ian Horrocks EMAIL Department of Computer Science, University of Oxford Parks Road, Oxford OX1 3QD, United Kingdom |
| Pseudocode | Yes | Algorithm 1: Greedy endomorphism checker. (Page 24) |
| Open Source Code | Yes | We have implemented our techniques in the PAGOd A system1 using RDFox as a datalog reasoner... 1. http://www.cs.ox.ac.uk/isg/tools/PAGOd A/ (Page 3) and We have implemented our approach in a system called PAGOd A, which is written in Java and is available under an academic license. (Page 23) |
| Open Datasets | Yes | LUBM and UOBM are widely-used reasoning benchmarks (Guo, Pan, & Heflin, 2005; Ma, Yang, Qiu, Xie, Pan, & Liu, 2006). (Page 27) and Ch EMBL, Reactome, and Uniprot are realistic ontologies that have been made publicly available through the European Bioinformatics Institute (EBI) linked data platform.10 http://www.ebi.ac.uk/rdf/platform (Page 28) and All test ontologies, queries, and results are available online.8 http://www.cs.ox.ac.uk/isg/tools/PAGOd A/2015/jair/ (Page 26). |
| Dataset Splits | No | For LUBM we used datasets of increasing size with a step of n = 100. For UOBM we also used increasingly large datasets with step n = 100 and we also considered a smaller step of n = 5 for hard queries. Finally, in the case of EBI s datasets, we implemented a data sampling algorithm based on random walks (Leskovec & Faloutsos, 2006) and computed subsets of the data of increasing sizes from 1% of the original dataset up to 100% in steps of 10%. This describes how dataset sizes were varied for scalability testing, not specific training/test/validation splits. |
| Hardware Specification | Yes | Experiments were conducted on a 32 core 2.60GHz Intel Xeon E5-2670 with 250GB of RAM, and running Fedora 20. (Page 26) |
| Software Dependencies | Yes | We have implemented our approach in a system called PAGOd A, which is written in Java... Our system integrates the datalog reasoner RDFox (Motik et al., 2014) and the fully-fledged OWL 2 reasoner Hermi T (Glimm et al., 2014) as black-boxes , and we also exploit the combined approach for ELHOr ? (see Section 4.2) implemented in KARMA (Stefanoni et al., 2014). (Page 23) |
| Experiment Setup | No | The paper describes the overall evaluation methodology, including the datasets used, how scalability was tested by varying dataset sizes (e.g., 'datasets of increasing size with a step of n = 100', 'computed subsets of the data of increasing sizes from 1%... up to 100% in steps of 10%'), and the test queries. However, it does not provide specific experimental setup details such as hyperparameter values, learning rates, batch sizes, optimizer settings, or other typical training configurations as would be expected for a machine learning model. |