Selective Explanations
Authors: Lucas Monteiro Paes, Dennis Wei, Flavio Calmon
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments on various models and datasets demonstrate that feature attributions via selective explanations strike a favorable balance between explanation quality and computational efficiency. This section analyzes the performance of selective explanations and its different components (i) uncertainty measures and (ii) explanations with initial guess. All results are showed in terms of MSE from target explanations, check Appendix D for the same results using Spearman s Rank Correlation. |
| Researcher Affiliation | Collaboration | Lucas Monteiro Paes Harvard University lucaspaes@g.harvard.edu Dennis Wei IBM Research dwei@us.ibm.com Flavio P. Calmon Harvard University flavio@seas.harvard.edu |
| Pseudocode | Yes | Algorithm 1 describes the procedure to compute the uncertainty metric, selection function, and combination function using the results we describe in Section 3 and 4. |
| Open Source Code | Yes | The code for generating selective explanations can be found at https://github.com/Lucas Monteiro Paes/selective-explanations. |
| Open Datasets | Yes | Datasets & Tasks: We use four datasets: two tabular datasets UCI-Adult [1] and UCI-News [8], and two text classification datasets Yelp Review [41] and Toxigen [14]. |
| Dataset Splits | Yes | We consider a dataset D = {(xi, yi)}N i=1 comprised of N > 0 samples divided into three parts: Dtrain for training h and the explainers, Dcal for calibration and validation, and Dtest for testing. Thus, D = Dtrain Dcal Dtest. The dataset D with N = 4000 samples was partitioned in three parts, Dtrain with 50% of points, Dcal with 25% of points, and Dtest with the other 25% of points. |
| Hardware Specification | Yes | All experiments were run in a A100 40 GB GPU. |
| Software Dependencies | No | High-quality and Monte Carlo explanations are computed using the Captum library [18]. Seaborn [36] is used to compute 95% confidence intervals using the bootstrap method. No specific version numbers for these or other critical software were found. |
| Experiment Setup | Yes | Models: For the tabular datasets, we train a multilayer perceptron [15] to learn the desired task. We use the Hugging Face Bert-based model textattack/bert-base-uncased-yelp-polarity [25] for the Yelp dataset and the Roberta-based model tomh/toxigen_roberta [14] for the Toxigen. Uncertainty metrics: we train k = 20 amortized explainers per task to compute the deep uncertainty. The dataset D with N = 4000 samples was partitioned in three parts, Dtrain with 50% of points, Dcal with 25% of points, and Dtest with the other 25% of points. |