Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Uncovering Unique Concept Vectors through Latent Space Decomposition
Authors: Mara Graziani, Laura O'Mahony, An-phi Nguyen, Henning Müller, Vincent Andrearczyk
TMLR 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our extensive experiments reveal that the majority of our concepts are readily understandable to humans, exhibit coherency, and bear relevance to the task at hand. Moreover, we showcase the practical utility of our method in dataset exploration, where our concept vectors successfully identify outlier training samples affected by various confounding factors. |
| Researcher Affiliation | Collaboration | Mara Graziani EMAIL Institute of Informatics University of Applied Sciences of Western Switzerland (Hes-so Valais) Sierre, CH 3960, Switzerland Laura O Mahony Laura EMAIL Department of Computer Science and Information Systems University of Limerick Limerick, V94 T9PX, Ireland An-Phi Nguyen EMAIL Biognosys Zurich, CH 8952, Switzerland |
| Pseudocode | No | The paper describes the method in three steps (variance alignment via SVD, ranking based on sensitivity, identification of unique concept vectors) using prose and mathematical formulas, but does not provide structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper mentions links for Open Review and for a human evaluation test (https://forms.gle/MJ63G984ERvozuF38), and cites a benchmark code (github.com/AI4LIFE-GROUP/Open XAI/blob/main/Open XAI%20quickstart.ipynb) which is external. However, it does not provide an explicit statement or a direct link to the source code for the methodology described in this paper. |
| Open Datasets | Yes | Specifically, we present the results obtained using Inception V3 (IV3) (Szegedy et al., 2016) with pretrained weights on the Image Net ILSVRC2012 dataset (Russakovsky et al., 2015). We consider a subset of ten dog breeds classes in Image Net, referred as Image Woof (Howard, 2019). Here we demonstrate the applicability of concept discovery to non-imaging applications such as COMPAS. |
| Dataset Splits | Yes | Figure 3D shows the accuracy drop observed when removing the concept directions from the first most important to the fifth, computed on the training images and 1000 validation images. |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU models, CPU models, or memory specifications used for running the experiments. It mentions "computational feasibility and accommodate our infrastructure capabilities" but no concrete specifications. |
| Software Dependencies | No | The paper does not provide specific software dependency details with version numbers (e.g., Python, PyTorch, TensorFlow, CUDA versions) needed to replicate the experiment. |
| Experiment Setup | Yes | After training IV3_noise0 from scratch for 100 epochs using stochastic gradient descent and standard parameters, the model achieved a remarkable test accuracy of 99.79% on images augmented with the squares. |