Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Explain Yourself, Briefly! Self-Explaining Neural Networks with Concise Sufficient Reasons
Authors: Shahaf Bassan, Ron Eliav, Shlomit Gur
ICLR 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We assess SST across image and language classification tasks, comparing the sufficient reasons generated by our approach to those from state-of-the-art post-hoc methods. Following common conventions (Wu et al. (2024b); Ignatiev et al. (2019); Bassan & Katz (2023)), we evaluate these subsets on three metrics: (i) the mean time taken to generate a sufficient reason, (ii) the mean size of the sufficient reasons, and (iii) the mean faithfulness, i.e., how sufficient the subsets are. Particularly, when evaluating faithfulness, we categorize the results into three distinct types of sufficiency: baseline, probabilistic, and robust faithfulness. 5.1 SUFFICIENT SUBSET TRAINING FOR IMAGES We train SST-based models on three prominent image classification tasks: MNIST (Deng (2012)) digit recognition using a feed-forward neural network, CIFAR-10 (Krizhevsky et al. (2009)) using Resnet18 (He et al. (2016)), and IMAGENET using a pre-trained Resnet50 model (He et al. (2016)). ... Table 1: SST with robust masking vs. post-hoc methods. Results depict average explanation size (%), 10-minute timeouts (%), average time (seconds), and robust faithfulness (%). |
| Researcher Affiliation | Collaboration | Shahaf Bassan1,2, Ron Eliav1,3, Shlomit Gur1 IBM Research1, The Hebrew University of Jerusalem2, Bar-Ilan University3 EMAIL |
| Pseudocode | No | The paper describes the methodology using mathematical formulations (e.g., equations 1-6) and descriptive text, but does not include any explicitly labeled pseudocode blocks or algorithms. |
| Open Source Code | Yes | Code is available at: https://github.com/IBM/SAX/tree/main/ICLR25. |
| Open Datasets | Yes | We train SST-based models on three prominent image classification tasks: MNIST (Deng (2012)) digit recognition using a feed-forward neural network, CIFAR-10 (Krizhevsky et al. (2009)) using Resnet18 (He et al. (2016)), and IMAGENET using a pre-trained Resnet50 model (He et al. (2016)). To simplify the training process, we set the threshold at τ = 0.5 and the faithfulness coefficient at λ = 1, conducting the grid search exclusively over the cardinality coefficient ξ. Full training details can be found in Appendix E. ... We assess our results using two prevalent language classification tasks: IMDB sentiment analysis (Maas et al. (2011)) and SNLI (Bowman et al. (2015)). All models are trained over a BERT base (Devlin et al. (2018)). |
| Dataset Splits | Yes | All image classification configurations underwent a grid search across various learning rates α, with values {10 2, 10 3, 10 4, 10 5, 10 6, 10 7} for both standard and SST training. For SST, additional grid searches were conducted for the cardinality coefficient ξ options: {10 5, 10 6, 10 7, 10 8, 10 9, 10 10, 10 11}. All models were trained using the Adam optimizer, a batch size of 64, and held-out validation and test sets. |
| Hardware Specification | Yes | The experiments were conducted using four Intel(R) Xeon(R) Gold 6258R @ 2.70GHz CPUs. For the standard training scenario, the optimal configuration selected was α := 10 4. ... We train a Res Net18 architecture (He et al. (2016)) (which is not pre-trained) as our base model using four Intel(R) Xeon(R) Gold 6258R @ 2.70GHz CPUs and one Nvidia A100-SXM4-80GB GPU. ... We train an SST-based model on a pre-trained Res Net50 (He et al. (2016)) for IMAGENET classification using nine Intel(R) Xeon(R) Gold 6258R @ 2.70GHz CPUs and one Nvidia A100-SXM4-80GB GPU. ... The training is conducted on 16 Intel(R) Xeon(R) Gold 6258R CPUs at 2.70GHz and one Nvidia A100-SXM4-80GB GPU. ... The setup included 16 Intel(R) Xeon(R) Gold 6258R CPUs at 2.70GHz and one Nvidia A100-SXM4-80GB GPU. |
| Software Dependencies | No | The paper mentions using 'Adam optimizer' and 'Adam W optimizer' and refers to a 'pre-trained Bert-base (Devlin et al. (2018)) model', but does not provide specific version numbers for these software components or the deep learning framework used (e.g., PyTorch, TensorFlow). |
| Experiment Setup | Yes | All image classification configurations underwent a grid search across various learning rates α, with values {10 2, 10 3, 10 4, 10 5, 10 6, 10 7} for both standard and SST training. For SST, additional grid searches were conducted for the cardinality coefficient ξ options: {10 5, 10 6, 10 7, 10 8, 10 9, 10 10, 10 11}. All models were trained using the Adam optimizer, a batch size of 64, and held-out validation and test sets. For robust masking, a PGD ℓ attack with ϵ := 0.12 was used, consisting of 10 steps each of size α := 10 2. ... The grid search focused on learning rates α := {2e 5, 3e 5, 5e 5}, which are the typical values used for optimizing pre-trained Bert models (Devlin et al. (2018)). For SST, an additional grid search was conducted for the cardinality coefficient ξ options: {10 4, 10 5, 10 6, 10 7, 10 8}. Optimization was performed using the standard Adam W optimizer, with a batch size of 32 and held-out validation and test sets. |