Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Causal Abstractions of Neural Networks
Authors: Atticus Geiger, Hanson Lu, Thomas Icard, Christopher Potts
NeurIPS 2021 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We apply this method in a case study to analyze neural models trained on Multiply Quantified Natural Language Inference (MQNLI) corpus, a highly complex NLI dataset that was constructed with a tree-structured natural logic causal model. |
| Researcher Affiliation | Academia | Atticus Geiger , Hanson Lu , Thomas Icard, and Christopher Potts Stanford Stanford, CA 94305-2150 EMAIL |
| Pseudocode | No | The paper does not contain any pseudocode or algorithm blocks. |
| Open Source Code | Yes | We provide tools for causal abstraction analysis at http://github.com/hansonhl/antra and the code base for this paper at http://github.com/atticusg/Interchange |
| Open Datasets | Yes | The Multiply Quantified NLI (MQNLI) dataset of Geiger et al. [10] contains templatically generated English-language NLI examples... |
| Dataset Splits | Yes | MQNLI has train/dev/test splits that vary in their difficulty. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory) used for running experiments. |
| Software Dependencies | No | The paper mentions BERT and Huggingface's transformers but does not provide specific version numbers for these or other software dependencies. |
| Experiment Setup | No | Additional model and training details are given in Appendix B. (This indicates details exist but are not in the provided main text). |