Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
Authors: Yair Ori Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, Roi Reichart
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our empirical results demonstrate the excellent performance of CF generation models as model-agnostic explainers. |
| Researcher Affiliation | Collaboration | T Faculty of Data and Decision Sciences, Technion, IIT CColumbia University Data Science Institute, MMicrosoft Research India |
| Pseudocode | No | The paper describes methods and objectives using prose and mathematical equations but does not include any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code: https://github.com/Yair Gat/causal-explanations |
| Open Datasets | Yes | We employ CEBa B (Abraham et al., 2022), the only established benchmark for evaluating causal explanation methods. |
| Dataset Splits | Yes | CEBa B contains two train sets, exclusive (N = 1463) and inclusive (which we do not use), development (N = 1672), and test (N = 1688) sets. |
| Hardware Specification | No | The paper mentions 'GPUs' in a general context but does not provide specific hardware details like exact GPU or CPU models used for experiments. |
| Software Dependencies | No | The paper mentions specific language models and APIs used (e.g., MPNet, Chat GPT, GPT-4) but does not provide version numbers for ancillary software libraries or frameworks required for replication. |
| Experiment Setup | Yes | We use τ = 0.1, train the causal representation models for 12 epochs... We use a learning rate of 5e-6 for training the causal representation models. We use a learning rate 1e-5 and a batch size 16 for the fine-tuned baselines, explained models and concept predictors. |