Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
Authors: Colin Wei, Kendrick Shen, Yining Chen, Tengyu Ma
ICLR 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This work provides a unified theoretical analysis of self-training with deep networks for semi-supervised learning, unsupervised domain adaptation, and unsupervised learning. At the core of our analysis is a simple but realistic expansion assumption... Our results help explain the empirical successes of recently proposed self-training algorithms which use input consistency regularization... In Section E.1, we provide details for the GAN experiment in Figure 1. We also provide empirical evidence for our theoretical intuition... In Section E.2, we perform ablation studies for pseudolabeling which show that components of our theoretical objective (4.1) do improve performance. Table 1 shows the performance of these methods on six unsupervised domain adaptation benchmarks. |
| Researcher Affiliation | Academia | Colin Wei & Kendrick Shen & Yining Chen & Tengyu Ma Department of Computer Science Stanford University Stanford, CA 94305, USA {colinwei,kshen6,cynnjjs,tengyuma}@stanford.edu |
| Pseudocode | No | The paper presents theoretical formulations and describes algorithmic procedures, but it does not include any explicitly labeled 'Pseudocode' or 'Algorithm' blocks. |
| Open Source Code | No | The paper references and links to a third-party 'robustness library' used in their experiments, but it does not provide an explicit statement or link to the source code for the methodology developed and described in this paper. |
| Open Datasets | Yes | We categorize images into 10 superclasses chosen in the robustness library of Engstrom et al. (2019)... We use 128 by 128 images sampled from a pre-trained Big GAN (Brock et al., 2018)... Table 1: Validation accuracy on the target data of various self-training methods. Source MNIST MNIST SVHN Syn Digits Syn Signs STL-10 Target SVHN MNIST-M MNIST SVHN GTSRB CIFAR-10... We use the same dataset setup and model architecture for each dataset as (Shu et al., 2018). |
| Dataset Splits | No | After self-training, the validation accuracy of new classifier e G improves to 95.69%... We use early stopping and display the best accuracy achieved during training... The value of the learning rate is tuned on the validation set for each dataset and method in the range of values {0.03, 0.01, 0.003, 0.001}... The paper explicitly refers to a 'validation set' and tuning on it, but it does not provide specific percentages, counts, or explicit descriptions of how the training/validation splits were created within the paper's text itself, only mentioning it follows another paper's setup. |
| Hardware Specification | No | The paper describes models like ResNet-56 and discusses training neural networks, implying the use of computational hardware. However, it does not specify any particular GPU, CPU, or other hardware models used for the experiments. |
| Software Dependencies | No | The paper mentions using the 'robustness library' of Engstrom et al. (2019) and VAT (Miyato et al., 2018), but it does not provide version numbers for these or any other software dependencies (e.g., deep learning frameworks, programming languages). |
| Experiment Setup | Yes | All classifiers are optimized using SGD with cosine learning rate and weight decay of 5e-4 and target batch size of 128. The value of the learning rate is tuned on the validation set for each dataset and method in the range of values {0.03, 0.01, 0.003, 0.001}. We choose λv, the coefficient of the VAT loss, by tuning in the same manner in the range {3, 10, 30}. For Min Ent+VAT+AMO, we fix the best hyperparameters for PL+VAT+AMO+Min Ent and tune λs {0.25, 0.5, 1} and fix λt = 1. We also tune the batch size for the source loss in {64, 128}. We use early stopping and display the best accuracy achieved during training. |