Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
AskWorld: Budget-Sensitive Query Evaluation for Knowledge-on-Demand
Authors: Mehdi Samadi, Partha Talukdar, Manuela Veloso, Tom Mitchell
IJCAI 2015 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through extensive experiments on real world datasets, we demonstrate Ask World s capability in selecting most informative resources to query within test-time constraints, resulting in improved performance compared to competitive baselines. |
| Researcher Affiliation | Academia | Mehdi Samadi Carnegie Mellon University EMAIL Partha Talukdar Indian Institute of Science EMAIL Manuela Veloso Carnegie Mellon University EMAIL Tom Mitchell Carnegie Mellon University EMAIL |
| Pseudocode | Yes | Algorithm 1 Ask World: Query Evaluation for Knowledge-on Demand |
| Open Source Code | No | The paper does not provide any specific links to source code or statements about its public release. |
| Open Datasets | Yes | For the experiments in this section, we use 25 categories randomly chosen from all the categories that are in common between Freebase [Bollacker et al., 2008] and NELL [Mitchell et al., 2015] knowledge bases. |
| Dataset Splits | No | The paper states: 'For each predicate, 200 random instances are provided as seed examples to train Ask World, and these are partitioned into two sets: classifier-training and policy-training.' and '50 instances are also randomly chosen as the test data for each predicate'. It does not explicitly mention a 'validation' split or cross-validation. |
| Hardware Specification | No | The paper does not provide specific details on the hardware used for running experiments (e.g., CPU/GPU models, memory specifications). |
| Software Dependencies | No | The paper mentions using Support Vector Machines (SVM) but does not provide specific software names with version numbers for any libraries, frameworks, or programming languages used. |
| Experiment Setup | Yes | For other parameters, we use a learning rate of 0.1, depth of 2 for each decision tree (depth of higher than 2 makes Greedy Miser inapplicable for small budget values), squared loss function, and a total of 300 regression trees in the final additive classifier. The result for Ask World (V*) is obtained by abstracting MDP using δ = 5, ordering queries using their information gain, and selecting top k% of features with non-zero information gain. In our experiments, we choose k = 50% which results in approximately 15M states in the MDP. |