Learning from the Best: Rationalizing Predictions by Adversarial Information Calibration
Authors: Lei Sha, Oana-Maria Camburu, Thomas Lukasiewicz13771-13779
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results on a sentiment analysis task as well as on three tasks from the legal domain show the effectiveness of our approach to rationale extraction. We experimentally evaluate our method on a sentiment analysis dataset with ground-truth rationale annotations, and on three tasks of a legal judgement prediction dataset, for which we conducted human evaluations of the extracted rationales. |
| Researcher Affiliation | Academia | Lei Sha,1 Oana-Maria Camburu,1,2 Thomas Lukasiewicz1,2 1Department of Computer Science, University of Oxford, UK 2Alan Turing Institute, London, UK {lei.sha, oana-maria.camburu, thomas.lukasiewicz}@cs.ox.ac.uk |
| Pseudocode | No | The detailed algorithm for training is given in Appendix D in the extended paper. |
| Open Source Code | No | The paper references source code for baselines ("We use their source code for the Bernoulli4 and Hard Kuma5 baselines.") but does not provide concrete access to the source code for the methodology *described in this paper* (their own Info Cal model). |
| Open Datasets | Yes | We use the Beer Advocate3 dataset (Mc Auley, Leskovec, and Jurafsky 2012). This dataset contains instances of human-written multi-aspect reviews on beers. We use the CAIL2018 dataset6 (Zhong et al. 2018) for three tasks on legal judgment prediction. The dataset consists of criminal cases published by the Supreme People s Court of China.7 |
| Dataset Splits | No | The paper mentions a "dev set" for model selection ("selected the models with the best recall on the dev set"), implying a validation split, but it does not provide specific percentages, sample counts, or citations for how the training, validation, and test sets were split in the main text. It states that "The detailed data preprocessing and experimental settings are given in Appendix B in the extended paper." |
| Hardware Specification | No | We also acknowledge the use of Oxford s Advanced Research Computing (ARC) facility, of the EPSRC-funded Tier 2 facility JADE (EP/P020275/1), and of GPU computing support by Scan Computers International Ltd. This mentions general GPU support and facilities but does not specify exact GPU models, CPU models, or detailed hardware configurations. |
| Software Dependencies | No | The paper does not provide specific ancillary software details with version numbers (e.g., programming language versions, library versions, or solver versions). |
| Experiment Setup | No | The detailed data preprocessing and experimental settings are given in Appendix B in the extended paper. The details of the choice of neural architecture for each module of our model, as well as the training setup are given in Appendix B in the extended paper. This indicates that the details are available, but not within the provided paper text itself. |