On the Impact of Knowledge Distillation for Model Interpretability
Authors: Hyeongrok Han, Siwon Kim, Hyun-Soo Choi, Sungroh Yoon
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conducted various quantitative and qualitative experiments and examined the results on different datasets, different KD methods, and according to different measures of interpretability. |
| Researcher Affiliation | Collaboration | 1Department of Electrical and Computer Engineering, Seoul National University, Seoul, Republic of Korea 2Department of Computer Science and Engineering, Seoul National University of Science and Technology, Seoul, Republic of Korea 3ZIOVISION Inc., Chuncheon, Republic of Korea 4Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Republic of Korea. |
| Pseudocode | Yes | We have included a pseudocode for obtaining concept detectors in Appendix C to facilitate the understanding of network dissection. |
| Open Source Code | Yes | The code is available at https: //github.com/Rok07/KD_XAI.git. |
| Open Datasets | Yes | All models were trained on the Image Net dataset (Russakovsky et al., 2015) |
| Dataset Splits | Yes | We generated 6,400 training and 1,600 test samples. |
| Hardware Specification | Yes | CPU: Intel(R) Xeon(R) Gold 6258R GPU: NVIDIA A40 48GB GDDR6 |
| Software Dependencies | Yes | CUDA version: 11.4 Library: Py Torch (Paszke et al., 2019) |
| Experiment Setup | Yes | The total number of epochs was 100. We used SGD optimization as the optimizer. We set the initial learning rate to 0.1 and divided it by ten every 30, 60, and 90 epoch for scheduling. We set the temperature to four. α, a hyper-parameter to determine the ratio of the correct answer to zt, was set to 0.5 for training. |