Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Dialog-to-Action: Conversational Question Answering Over a Large-Scale Knowledge Base
Authors: Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, Jian Yin
NeurIPS 2018 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct experiments on a large-scale dataset [17] for conversation question answering, which consists of 200K dialogs with 1.6M turns over 12.8M entities from Wikidata. Results verify the bene๏ฌts of modeling dialog memory, and show that our semantic parsing-based approach outperforms a memory network based encoder-decoder model by a huge margin. |
| Researcher Affiliation | Collaboration | 1 The School of Data and Computer Science, Sun Yat-sen University. Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, P.R.China 2 Microsoft Research Asia, Beijing, China |
| Pseudocode | No | The paper describes the proposed model and algorithms in text but does not include any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | No | The paper does not provide any statements about making the source code available or include links to a code repository. |
| Open Datasets | Yes | We conduct the experiment on the CSQA dataset3. The dataset is created based on Wikidata4, including 152K dialogs for training, and 16K/28K dialogs for development/testing. 3https://amritasaha1812.github.io/CSQA/ |
| Dataset Splits | Yes | We conduct the experiment on the CSQA dataset3. The dataset is created based on Wikidata4, including 152K dialogs for training, and 16K/28K dialogs for development/testing. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used to run the experiments, such as GPU models, CPU types, or memory specifications. |
| Software Dependencies | No | The paper mentions software components and architectures like GRU and RNNs but does not provide specific version numbers for any software dependencies, libraries, or frameworks used. |
| Experiment Setup | Yes | We use a batch size of 32 for the training process and use Adam optimizer. The initial learning rate is set to 0.001. We use beam search at the inference phase. |