Explaining Image Classifiers by Counterfactual Generation
Authors: Chun-Hao Chang, Elliot Creager, Anna Goldenberg, David Duvenaud
ICLR 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We first evaluate the various infilling strategies and objective functions for FIDO. We then compare explanations under several classifier architectures. In section 4.5 we show that FIDO saliency maps outperform BBMP (Fong & Vedaldi, 2017) in a successive pixel removal task where pixels are in-filled by a generative model (instead of set to the heuristic value). FIDO also outperforms the method from (Dabkowski & Gal, 2017) on the so-called Saliency Metric on Image Net. |
| Researcher Affiliation | Academia | Chun-Hao Chang, Elliot Creager, Anna Goldenberg, & David Duvenaud University of Toronto, Vector Institute {kingsley,creager,duvenaud}@cs.toronto.edu anna.goldenberg@utoronto.ca |
| Pseudocode | Yes | Figure 5: Pseudo code comparison. Differences between the approaches are shown in blue. Algorithm 1: BBMP (Fong & Vedaldi, 2017) ... Algorithm 2: FIDO (Ours) |
| Open Source Code | Yes | We released the code in Py Torch at https://github. com/zzzace2000/FIDO-saliency. |
| Open Datasets | Yes | By leveraging a powerful in-filling conditional generative model we produce saliency maps on Image Net that identify relevant and concentrated pixels better than existing methods. We evaluate on Res Net and carry out our scoring procedure on 1, 533 randomly-selected correctly-predicted Image Net validation images... |
| Dataset Splits | Yes | We evaluate on Res Net and carry out our scoring procedure on 1, 533 randomly-selected correctly-predicted Image Net validation images... Using FIDO with various infilling methods, we report the average error rate across all 50, 000 validation images in Table 1. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, memory) were provided. The paper only mentions that its Py Torch implementation 'takes about one minute on a single GPU to finish one image'. |
| Software Dependencies | No | No specific software version numbers were provided. The paper mentions 'Py Torch implementation' and optimization using 'Adam (Kingma & Ba, 2014)' but without version details for PyTorch or other libraries. |
| Experiment Setup | Yes | We optimize using Adam (Kingma & Ba, 2014) with learning rate 0.05 and linearly decay the learning rate for 300 batches in all our experiments. We initialize all our dropout rates θ to 0.5 since we find it increases the convergence speed and avoids trivial solutions. We use temperature 0.1. We select λ = 1e 3 for BBMP and FIDO. We found unsatisfactory results for batch size less than 4. |