Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning
Authors: Yuncheng Hua, Yuan-Fang Li, Gholamreza Haffari, Guilin Qi, Wei Wu
IJCAI 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluated our MARL model on the CQA dataset [Saha et al., 2018]. CQA is a large-scale complex question answering dataset containing 944K/100K/156K question-answer pairs for training/validation/test... Our full model MARL achieves the best overall performance of 66.96% and 77.71% for macro and micro F1, respectively, outperforming all the baseline models KVmem, CIPITR-All, and CIPITR-Sep. (Tables 1 and 2 also present experimental results and an ablation study). |
| Researcher Affiliation | Academia | 1School of Computer Science and Engineering, Southeast University, Nanjing, China 2Faculty of Information Technology, Monash University, Melbourne, Australia 3Southeast University-Monash University Joint Research Institute, Suzhou, China 4Key Laboratory of Computer Network and Information Integration, Southeast University, China |
| Pseudocode | Yes | Algorithm 1: The MARL algorithm |
| Open Source Code | Yes | We have released our code at https://github.com/Devin Jake/MARL. |
| Open Datasets | Yes | We evaluated our MARL model on the CQA dataset [Saha et al., 2018]. CQA is a large-scale complex question answering dataset containing 944K/100K/156K question-answer pairs for training/validation/test. |
| Dataset Splits | Yes | CQA is a large-scale complex question answering dataset containing 944K/100K/156K question-answer pairs for training/validation/test. |
| Hardware Specification | No | The paper mentions 'We implemented the MARL model in Py Torch' but does not provide any specific hardware details such as GPU models, CPU types, or memory specifications used for running experiments. |
| Software Dependencies | No | The paper states 'We implemented the MARL model in Py Torch' and 'Adam optimizer was applied in RL' but does not specify version numbers for PyTorch, Adam, or any other software dependencies. |
| Experiment Setup | Yes | We set η1 = 1e-4 when adapting the model to each new task, and set η2 = 0.1 to optimize θ with the gradient update derived from the meta-test data. The reward that the adaptive programmer gained was used to update the retriever parameter φ through the Ada Bound optimizer [Luo et al., 2019] in which the learning rate η3 was initially set to 1e-3 and the final (SGD) learning rate was set to 0.1. When finding the top-N support set, we set N = 5. ... We trained the MARL model with the batch size of 1 and stopped training when the accuracy on the validation set converged (at around 30 epochs). |