Question Answering via Integer Programming over Semi-Structured Knowledge
Authors: Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, Dan Roth
IJCAI 2016 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | On a dataset of real, unseen science questions, our system significantly outperforms (+14%) the best previous attempt at structured reasoning for this task, which used Markov Logic Networks (MLNs). |
| Researcher Affiliation | Collaboration | University of Illinois at Urbana-Champaign, IL, U.S.A. {khashab2,danr}@illinois.edu Allen Institute for Artificial Intelligence (AI2), Seattle, WA, U.S.A. {tushark,ashishs,peterc,orene}@allenai.org |
| Pseudocode | No | The paper describes its ILP formulation and variables but does not present any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | Yes | Table Corpus and the ILP model are available at allenai.org. |
| Open Datasets | Yes | We use the same question set as Clark et al. [2016], which consists of all non-diagram multiple-choice questions from 12 years of the NY Regents 4th Grade Science exams. |
| Dataset Splits | Yes | The set is split into 108 development questions and 129 hidden test questions based on the year they appeared in (6 years each). |
| Hardware Specification | Yes | For evaluations, we use a 2-core 2.5 GHz Amazon EC2 linux machine with 16 GB RAM. |
| Software Dependencies | No | The paper mentions the use of the "open source SCIP engine [Achterberg, 2009]" and refers to "Word Net-based" functions, but it does not specify version numbers for these or other software components. |
| Experiment Setup | Yes | Given a question Q, we select the top 7 tables from the Table Corpus using the the standard TF-IDF score of Q with tables treated as bag-of-words documents. For each selected table, we choose the 20 rows that overlap with Q the most. |