Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation
Authors: Yichen Zhu, Ning Liu, Zhiyuan Xu, Xin Liu, Weibin Meng, Louis Wang, Zhicai Ou, Jian Tang
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We validate the effectiveness of our approach on multiple datasets with varying network architectures. In all settings, our proposed method is able to exceed the performance of competitive state-of-the-art techniques. |
| Researcher Affiliation | Collaboration | Yichen Zhu, Midea Group Ning Liu Midea Group Zhiyuan Xu Midea Group Xin Liu East China Normal University Weibing Meng Tsinghua University Yi Wang Midea Group Zhicai Ou Midea Group Jian Tang Midea Group |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | No | The code is proprietary. |
| Open Datasets | Yes | To this end, we investigate the existence of the undistillable classes over three standard datasets (CIFAR100 [21], Image Net1K [6], CUB-200 [42]) with more than 20 modern distillation techniques. |
| Dataset Splits | Yes | The teaching curve is calculated on the validation set. |
| Hardware Specification | No | The main body of the paper does not specify the hardware (e.g., GPU models, CPU types) used for experiments. While the self-assessment checklist states 'Yes' for hardware specification, this information is not present in the provided paper text. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers. |
| Experiment Setup | No | The paper states 'We refer reader to the Appendix for the implementation details.' As the appendix is not provided, the specific experimental setup details (e.g., hyperparameters, optimizer settings) are not available in the main text. |