Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
Authors: Wenhan Xiong, Xiang Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, Barlas Oguz
ICLR 2021 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We propose a simple and efficient multi-hop dense retrieval approach for answering complex open-domain questions, which achieves state-of-the-art performance on two multi-hop datasets, Hotpot QA and multi-evidence FEVER. Our experiments focus on two datasets: Hotpot QA and Multi-evidence FEVER. |
| Researcher Affiliation | Collaboration | 1University of California, Santa Barbara 2University of Massachusetts Amherst Facebook AI University College London EMAIL, EMAIL, EMAIL |
| Pseudocode | No | The paper does not contain any pseudocode or algorithm blocks. |
| Open Source Code | Yes | 1https://github.com/facebookresearch/multihop_dense_retrieval. |
| Open Datasets | Yes | Our experiments focus on two datasets: Hotpot QA and Multi-evidence FEVER. Hotpot QA (Yang et al., 2018) includes 113k multi-hop questions... Multi-evidence FEVER includes 20k claims from the FEVER (Thorne et al., 2018) fact verification dataset... |
| Dataset Splits | Yes | Our experiments focus on two datasets: Hotpot QA and Multi-evidence FEVER. Hotpot QA (Yang et al., 2018) includes 113k multi-hop questions... We conduct further analysis on Hotpot QA dev. |
| Hardware Specification | Yes | All the experiments are conducted on a machine with 8 32GB V100 GPUs. |
| Software Dependencies | No | The paper mentions software like 'Huggingface Transformers' and 'FAISS' but does not specify their version numbers, which is required for reproducibility. |
| Experiment Setup | Yes | Table 11: Hyperparameters of Retriever [lists learning rate 2e-5, batch size 150, etc.] Table 12: Hyperparameters of Extractive Reader (ELECTRA) [lists learning rate 5e-5, batch size 128, etc.] |