Faithful Embeddings for Knowledge Base Queries
Authors: Haitian Sun, Andrew Arnold, Tania Bedrax Weiss, Fernando Pereira, William W. Cohen
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate Em QL first intrinsically for its ability to model set expressions [10], and then extrinsically as the reasoning component in two multi-hop KB question answering benchmarks (KBQA). The numbers are shown in Table 2. Following the Query2Box paper [17] we use d = 400 for their model and report Hits@3 (see Supplementary Materials D for other metrics). To see if larger embeddings would improve Q2B s performance on entailment tasks, we increased the dimension size to d = 2000, and observed a decrease in performance, relative to the tuned value d = 400 [17]. |
| Researcher Affiliation | Industry | Haitian Sun Andrew O. Arnold Tania Bedrax-Weiss Fernando Pereira William W. Cohen Google Research {haitiansun,tbedrax,pereira,wcohen}@google.com AWS AI anarnld@amazon.com |
| Pseudocode | No | The paper describes algorithmic steps in prose, but does not include any clearly labeled 'Pseudocode' or 'Algorithm' blocks. |
| Open Source Code | Yes | Code available at https://github.com/google-research/language |
| Open Datasets | Yes | The Meta QA datasets [29] contain multi-hop questions in the movie domain that are answerable using the Wiki Movies KB [13]... The Web Questions SP model. This dataset [27] contains 4,737 natural language questions generated from Freebase. |
| Dataset Splits | Yes | To evaluate performance for QE, Ren et al. first hold out some triples from the KB for validation and test, and take the remaining triples as the training KB. |
| Hardware Specification | No | Here d = 2000 was the largest value of d supported by our GPUs. (Mentions 'GPUs' but no specific models or other hardware details). |
| Software Dependencies | No | No specific software dependencies with version numbers (e.g., 'Python 3.8', 'PyTorch 1.9', 'CUDA 11.1') are provided. |
| Experiment Setup | Yes | For Em QL, we use d = 64, k = 1000, NW = 2000 and ND = 20 throughout. |