CoSy: Evaluating Textual Explanations of Neurons
Authors: Laura Kopf, Philine L Bommer, Anna Hedström, Sebastian Lapuschkin, Marina Höhne, Kirill Bykov
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We introduce COSY (Concept Synthesis), a novel, architecture-agnostic framework for evaluating textual explanations of latent neurons. Given textual explanations, our proposed framework uses a generative model conditioned on textual input to create data points representing the explanations. By comparing the neuron s response to these generated data points and control data points, we can estimate the quality of the explanation. We validate our framework through sanity checks and benchmark various neuron description methods for Computer Vision tasks, revealing significant differences in quality. |
| Researcher Affiliation | Academia | 1TU Berlin, Germany 2BIFOLD, Germany 3UMI Lab, ATB Potsdam, Germany 4Fraunhofer Heinrich-Hertz-Institute, Germany 5University of Potsdam, Germany |
| Pseudocode | No | The paper describes the COSY framework in three steps (Generate Synthetic Data, Collect Neuron Activations, Score Explanations) with textual descriptions and mathematical formulas, but it does not include a formally structured pseudocode or algorithm block. |
| Open Source Code | Yes | We provide an open-source implementation on Git Hub1. 1https://github.com/lkopf/cosy |
| Open Datasets | Yes | The Image Net dataset focuses on objects, whereas the Places365 dataset is designed for scene recognition. and For evaluation with COSY, we use the corresponding validation datasets the models were pre-trained on as the control dataset. (References [36] and [42] are provided for ImageNet and Places365 respectively). |
| Dataset Splits | No | For generating explanations with the explanation methods, we use a subset of 50,000 images from the training dataset on which the models were trained. For evaluation with COSY, we use the corresponding validation datasets the models were pre-trained on as the control dataset. (While it mentions training and validation datasets, it does not provide specific numerical splits or detailed methodology for how these splits were partitioned for their experiments, only that they use existing validation sets as control data). |
| Hardware Specification | Yes | For running the task of image generation for COSY we use distributed inference across multiple GPUs with Py Torch Distributed, enabling image generation with multiple prompts in parallel. We run our script on three Tesla V100S-PCIE-32GB GPUs in an internal cluster. |
| Software Dependencies | No | For running the task of image generation for COSY we use distributed inference across multiple GPUs with Py Torch Distributed, enabling image generation with multiple prompts in parallel. (While PyTorch Distributed is mentioned, a specific version number for PyTorch or other libraries is not provided.) |
| Experiment Setup | Yes | For our analysis, we used only open-source and freely available text-to-image models, namely Stable Diffusion XL 1.0-base (SDXL) [34] and Stable Cascade (SC) [35]. We also varied the prompts for image generation. and If not stated otherwise, for all following experiments, Prompt 5 together with SDXL model was employed for image generation. and For INVERT we set the compositional length of the explanation as L = 1, where L N. and We generated 50 images per concept for 50 randomly chosen neurons from the avgpool layer of Res Net18. |