Re-Evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
Authors: Ananya B. Sai, Mithun Das Gupta, Mitesh M. Khapra, Mukundhan Srinivasan6220-6227
AAAI 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We design a battery of targeted attacks at the neural network based ADEM evaluation system and show that automatic evaluation of dialogue systems still has a long way to go. We report experiments on several such adversarial scenarios that draw out counterintuitive scores on the dialogue responses. We provide an analysis of this phenomenon and verify the claim experimentally as well. |
| Researcher Affiliation | Collaboration | Ananya B. Sai,1,2,4 Mithun Das Gupta,3 Mitesh M. Khapra,1,2 Mukundhan Srinivasan4 1Department of Computer Science and Engineering, Indian Institute of Technology, Madras 2Robert Bosch Center for Data Sciences and AI (RBC-DSAI), Indian Institute of Technology, Madras 3Microsoft, India 4NVIDIA, India |
| Pseudocode | No | The paper describes methods and processes in narrative text and tables but does not include any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide a specific link or statement about open-sourcing the code for the methodology or analysis described within this paper. It mentions using libraries like NLTK and Annoy, which are third-party tools. |
| Open Datasets | Yes | This analysis is presented in Table 3 using the Microsoft Research Social Media Conversation Corpus by Sordoni et al. (2015). For the following section of human constructed attacks, we run the experiments on the Microsoft Research Social Media Conversation Corpus (Sordoni et al. 2015). |
| Dataset Splits | No | The paper describes using the Microsoft Research Social Media Conversation Corpus for analysis and experiments but does not provide specific train, validation, or test dataset splits for their experimental setup. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., CPU, GPU models, memory, or cloud instance types) used for running the experiments. It only mentions "compute resources" in acknowledgements. |
| Software Dependencies | No | The paper mentions using "NLTK (Natural Language Tool Kit) library of python" and "Word Net from NLTK" but does not provide specific version numbers for these software components. |
| Experiment Setup | No | The paper focuses on evaluating an existing model (ADEM) and does not describe the specific hyperparameters or system-level training settings for its own experimental analysis. |