Knowledge Base Question Answering by Case-based Reasoning over Subgraphs
Authors: Rajarshi Das, Ameya Godbole, Ankita Naik, Elliot Tower, Manzil Zaheer, Hannaneh Hajishirzi, Robin Jia, Andrew Mccallum
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show that CBR-SUBG can answer queries requiring subgraph reasoning patterns and performs competitively with the best models on several KBQA benchmarks. Our subgraph collection strategy also produces more compact subgraphs (e.g. 55% reduction in size for Web QSP while increasing answer recall by 4.85%)1. (...) In this section, we demonstrate the effectiveness of the semiparametric approach of CBR-SUBG and show that the nonparametric and parametric component offer complementary strengths. For example, we show that the model performance improves as more evidence is dynamically retrieved by the nonparametric component ( 4.3). Similarly, CBR-SUBG can handle queries requiring reasoning patterns more complex than simple chains (i.e. subgraphs) because of the inductive bias provided by GNNs ( 4.1). It can handle new and unseen entities because of the sparse entity input features as a part of its design ( 4.1). We also show that the nonparametric subgraph selection of CBR-SUBG allows us to operate over a massive real-world KG (full Freebase KG) and obtain very competitive performance on several benchmark datasets including Web Questions SP (Yih et al., 2016), Freebase QA (Jiang et al., 2019) and Meta QA (Zhang et al., 2018). |
| Researcher Affiliation | Collaboration | Rajarshi Das * 1 Ameya Godbole * 2 Ankita Naik 3 Elliot Tower 3 Robin Jia 2 Manzil Zaheer 4 Hannaneh Hajishirzi 1 Andrew Mc Callum 3 1University of Washington 2University of Southern California 3UMass Amherst 4Google Deep Mind. |
| Pseudocode | No | The paper describes procedures and algorithms in prose but does not include any formal pseudocode blocks or algorithms labeled as such. |
| Open Source Code | Yes | Code, model, and subgraphs are available at https://github.com/rajarshd/CBR-SUBG |
| Open Datasets | Yes | We test the performance of CBR-SUBG on various KBQA benchmarks Meta QA (Zhang et al., 2018), Web QSP (Yih et al., 2016) and Freebase QA (Jiang et al., 2019). Meta QA comes with its own KB. For other datasets, the underlying KB is the full Freebase KB containing over 45 million entities (nodes) and 3 billion facts (edges). Please refer to the appendix for details about each dataset ( C). (...) Table 7 summarizes the basic statistics of the datasets used in our experiments. |
| Dataset Splits | Yes | For Meta QA, we use 3 GCN layers with GCN layer dimension of 32. For training we have used 5 nearest neighbors and 10 are used for evaluation for the 1-hop, 2-hop and 3-hop queries. (...) We generate 1000 graphs in each of the training, validation and test sets. (...) For each of the 15 graphs that share a common pattern type, we assign the 5 graphs that were put in the train set as the k NN queries. (...) Table 7 summarizes the basic statistics of the datasets used in our experiments: Dataset Train Dev Test Meta QA 1-hop 96,106 9,992 9,947 Meta QA 2-hop 118,980 14,872 14,872 Meta QA 3-hop 114,196 14,274 14,274 Web QSP 2,848 250 1,639 Freebase QA 20,358 2308 3996 |
| Hardware Specification | No | The paper does not provide specific details on the hardware used, such as GPU models, CPU types, or memory. It mentions that the subgraph collection strategy allows fitting into "GPU memory" but gives no further specifics. |
| Software Dependencies | No | The paper mentions "large pre-trained language models (Liu et al., 2019)", "ROBERTA-base encoder", and "Adam Optimizer" and "multi-relational R-GCN model model (Schlichtkrull et al., 2018)" but does not provide specific version numbers for these software components or frameworks. |
| Experiment Setup | Yes | For Meta QA, we use 3 GCN layers with GCN layer dimension of 32. For training we have used 5 nearest neighbors and 10 are used for evaluation for the 1-hop, 2-hop and 3-hop queries.We optimize the loss using Adam Optimizer with beta1 of 0.9, beta2 of 0.999 and epsilon of 1e-8. As well as the learning rate is set to be 0.00099 with temperature value of 0.0382 (1-hop), 0.0628 (2-hop) ,0.0779 (3-hop). All the models are trained for 5 epochs. Similarly for Web QSP, we use 3 GCN layers with GCN layer dimension of 32. But for training we used 10 nearest neighbors and 5 are used for evaluation. We optimize the loss using Adam Optimizer with beta1 of 0.9, beta2 of 0.999 and epsilon of 1e-8. As well as a learning rate of 0.0024 and temperature of 0.0645 is used. The model is trained for about 30 epochs. All hyper-parameters can also be found in our code-base. |