Case-Based Reasoning with Language Models for Classification of Logical Fallacies
Authors: Zhivar Sourati, Filip Ilievski, Hông-Ân Sandlin, Alain Mermoud
IJCAI 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments in in-domain and out-of-domain settings indicate that Case-Based Reasoning improves the accuracy and generalizability of language models. Our ablation studies suggest that representations of similar cases have a strong impact on the model performance, that models perform well with fewer retrieved cases, and that the size of the case database has a negligible effect on the performance. |
| Researcher Affiliation | Collaboration | 1Information Sciences Institute, University of Southern California, Marina del Rey, CA, USA 2Department of Computer Science, University of Southern California, Los Angeles, CA, USA 3Cyber-Defence Campus, armasuisse Science and Technology, Switzerland |
| Pseudocode | No | The paper describes the components of the CBR pipeline but does not provide any formal pseudocode or algorithm blocks. |
| Open Source Code | Yes | We make our code and data available to support future research on logical fallacy classification.1 1https://github.com/zhpinkman/CBR |
| Open Datasets | Yes | We use two logical fallacy datasets from [Jin et al., 2022], called LOGIC and LOGIC Climate. ...As LOGIC dataset is severely imbalanced, we augment its train split using two techniques, i.e., back-translation, and substitution of entities in the arguments with their synonymous terms. |
| Dataset Splits | No | The paper mentions 'train split' and 'test' datasets (LOGIC and LOGIC Climate) but does not explicitly detail a validation split or its size/proportion for reproduction. |
| Hardware Specification | No | The paper does not provide any specific hardware details used for running its experiments, such as GPU/CPU models or memory. |
| Software Dependencies | No | The paper mentions software like Sim CSE, BERT, RoBERTa, and ELECTRA, but does not provide specific version numbers for these or other software dependencies. |
| Experiment Setup | Yes | We use Sim CSE [Gao et al., 2021], a transformer-based retriever that is optimized for capturing overall sentence similarity, to compute the similarity between cases ( 2) and also use H = 8 heads for the multi-headed attention component. The depth of our classifier is d = 2. It uses gelu [Hendrycks and Gimpel, 2016] as an activation function. We analyze the performance of our model using k {1, 2, 3, 4, 5}. To test the generalization of our model with sparser case databases, we experiment with various ratios of the case database within {0.1, 0.4, 0.7, 1.0}. |