Spatial-Channel Token Distillation for Vision MLPs
Authors: Yanxi Li, Xinghao Chen, Minjing Dong, Yehui Tang, Yunhe Wang, Chang Xu
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments on Image Net for several MLP-like architectures demonstrate that the proposed token distillation mechanism can efficiently improve the accuracy. |
| Researcher Affiliation | Collaboration | 1School of Computer Science, University of Sydney, Australia 2Huawei Noah s Ark Lab 3School of Artificial Intelligence, Peking University. |
| Pseudocode | Yes | Algorithm 1 The mutual information regularization on spatial-channel tokens. |
| Open Source Code | No | The paper does not include an explicit statement or link indicating the availability of source code for the described methodology. |
| Open Datasets | Yes | Datasets. We use the Image Net-1K (Russakovsky et al., 2015) dataset for both distillation and evaluation. It has 1.3 million images covering 1,000 classes. |
| Dataset Splits | No | The paper mentions using ImageNet-1K for distillation and evaluation, but does not provide specific percentages or counts for training, validation, and test splits, nor does it cite predefined splits in a way that specifies the exact partitioning used. |
| Hardware Specification | Yes | The throughput is tested on Image Net-1K with 8 NVIDIA V100 GPUs and is listed in Table 1. |
| Software Dependencies | No | The paper does not specify version numbers for any key software components or libraries used. |
| Experiment Setup | No | The paper discusses distillation settings and mentions some architectural parameters (e.g., 512 dimensions for MINE network) but does not provide concrete hyperparameter values such as learning rate, batch size, number of epochs, or specific optimizer settings. |