Weighted Distillation with Unlabeled Examples
Authors: Fotis Iliopoulos, Vasilis Kontonis, Cenk Baykal, Gaurav Menghani, Khoa Trinh, Erik Vee
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate significant improvements on popular academic datasets and we accompany our results with a theoretical analysis which rigorously justifies the performance of our method in certain settings. |
| Researcher Affiliation | Industry | Fotis Iliopoulos Google Research fotisi@google.com Vasilis Kontonis Google Research kontonis@google.com Cenk Baykal Google Research baykalc@google.com Gaurav Menghani Google Research gmenghani@google.com Khoa Trinh Google Research khoatrinh@google.com Erik Vee Google Research erikvee@google.com |
| Pseudocode | Yes | We describe our method below and more formally in Algorithm 1 in Appendix A. The pseudocode for our method can be found in Algorithm 2 in Appendix A. |
| Open Source Code | No | The paper states 'Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No]' in Section 6, point 3a. |
| Open Datasets | Yes | SVHN [34] is an image classification dataset... CIFAR-10 and CIFAR-100 [24] are image classification datasets... Celeb A [15] is a large-scale face attributes dataset... Image Net [41] is a large-scale image classification dataset... |
| Dataset Splits | Yes | Our experiments are of the following form. The academic dataset we use each time is first split into two parts A and B. Part A, which is typically smaller, is used as the labeled dataset Sℓwhere the teacher model is trained on (recall the setting we described in Section 2.3). Part B is randomly split again into two parts which represent the unlabeled dataset Su and validation dataset Sv, respectively. ... In each experiment we use the first N {7500, 10000, 12500, 15000, 17500, 20000} examples as the labeled dataset Sℓ, and then the rest 73257 N images are randomly split to a labeled validation dataset Sv of size 2000, and an unlabeled dataset Su of size 71257 N. ... We use a validation set of 2000 examples. |
| Hardware Specification | Yes | We implemented all algorithms in Python making use of the Tensor Flow deep learning library [1]. We use 64 Cloud TPU v4s each with two cores. |
| Software Dependencies | No | The paper mentions 'Tensor Flow deep learning library [1]' but does not specify a version number for TensorFlow or any other software dependencies. |
| Experiment Setup | Yes | We always choose the temperature in the softmax of the models to be 1 for simplicity and consistency, and our metric for confidence is always the margin-score. Implementation details for our experiments and additional results can be found in Appendices B and C. ... In each experiment we use the first N {7500, 10000, 12500, 15000, 17500, 20000} examples as the labeled dataset Sℓ, and then the rest 73257 N images are randomly split to a labeled validation dataset Sv of size 2000, and an unlabeled dataset Su of size 71257 N. |