Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
Authors: Yair Ori Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, Roi Reichart
ICLR 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our empirical results demonstrate the excellent performance of CF generation models as model-agnostic explainers. |
| Researcher Affiliation | Collaboration | T Faculty of Data and Decision Sciences, Technion, IIT CColumbia University Data Science Institute, MMicrosoft Research India |
| Pseudocode | No | The paper describes methods and objectives using prose and mathematical equations but does not include any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code: https://github.com/Yair Gat/causal-explanations |
| Open Datasets | Yes | We employ CEBa B (Abraham et al., 2022), the only established benchmark for evaluating causal explanation methods. |
| Dataset Splits | Yes | CEBa B contains two train sets, exclusive (N = 1463) and inclusive (which we do not use), development (N = 1672), and test (N = 1688) sets. |
| Hardware Specification | No | The paper mentions 'GPUs' in a general context but does not provide specific hardware details like exact GPU or CPU models used for experiments. |
| Software Dependencies | No | The paper mentions specific language models and APIs used (e.g., MPNet, Chat GPT, GPT-4) but does not provide version numbers for ancillary software libraries or frameworks required for replication. |
| Experiment Setup | Yes | We use τ = 0.1, train the causal representation models for 12 epochs... We use a learning rate of 5e-6 for training the causal representation models. We use a learning rate 1e-5 and a batch size 16 for the fine-tuned baselines, explained models and concept predictors. |