PRIME: Prioritizing Interpretability in Failure Mode Extraction
Authors: Keivan Rezaei, Mehrdad Saberi, Mazda Moayeri, Soheil Feizi
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through several experiments on different datasets, we show that our method successfully identifies failure modes and generates high-quality text descriptions associated with them. These results highlight the importance of prioritizing interpretability in understanding model failures. |
| Researcher Affiliation | Academia | 1Department of Computer Science, University of Maryland {krezaei,msaberi,mmoayeri,sfeizi}@umd.edu |
| Pseudocode | No | The paper describes algorithms like 'Exhaustive Search' and 'Greedy Search' in paragraph text but does not present them in a structured pseudocode or algorithm block. |
| Open Source Code | No | The paper does not contain any explicit statements or links indicating that the source code for the described methodology is publicly available. |
| Open Datasets | Yes | We run experiments on models trained on Living17, Non Living26, Entity13 (Santurkar et al., 2020), Waterbirds (Sagawa et al., 2019), and Celeb A (Liu et al., 2015) (for age classification). |
| Dataset Splits | Yes | In this section, we inspect the effect of different hyperparameters (s, a) on the result of our method. ... Figure 10 shows the generalization plot over different datasets with respect to different hyperparameters. In Table 4, we also report correlation coefficien between train drop and test drop of failure modes over different datasets and values of s and a. |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU or CPU models, or any other computing infrastructure used for running experiments. |
| Software Dependencies | No | The paper mentions models like ResNet50 and DINO, and tools like RAM and CLIP ViT-B/16, and refers to 'Chat GPT 3.5, August 3 version' for generating synthetic images. However, it does not specify versions for core software libraries or frameworks (e.g., Python, PyTorch, TensorFlow) used in the main experimental setup or for reproducible ancillary software. |
| Experiment Setup | Yes | For Living17, Entity13, and Non Living26, we utilize a DINO self-supervised model (Caron et al., 2021) with Res Net50 backbone and fine-tune it over Living17. we used SGD with following hyperparameters to finetune the model for 5 epochs. lr = 0.001 momentum = 0.9 |