RoPAWS: Robust Semi-supervised Representation Learning from Uncurated Data
Authors: Sangwoo Mo, Jong-Chyi Su, Chih-Yao Ma, Mido Assran, Ishan Misra, Licheng Yu, Sean Bell
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate that Ro PAWS significantly improves PAWS for uncurated Semi-i Nat by +5.3% and curated Image Net by +0.4%. |
| Researcher Affiliation | Collaboration | Sangwoo Mo1 Jong-Chyi Su2 Chih-Yao Ma3 Mahmoud Assran3,4,5 Ishan Misra3 Licheng Yu3 Sean Bell3 1KAIST 2NEC Laboratories America 3Meta AI 4Mc Gill University 5Mila |
| Pseudocode | No | The paper describes its method using equations and textual explanations but does not include a formal pseudocode block or algorithm. |
| Open Source Code | Yes | Code: https://github.com/facebookresearch/suncet |
| Open Datasets | Yes | We follow the default setup of PAWS: Image Net (Deng et al., 2009) setup on Res Net-50 (He et al., 2016) for large-scale experiments, and CIFAR-10 (Krizhevsky et al., 2009) setup on WRN-28-2 (Zagoruyko & Komodakis, 2016) for small-scale experiments. Semi-i Nat (Su & Maji, 2021b) is a realistic large-scale benchmark for semi-supervised learning |
| Dataset Splits | Yes | We use 1% of labels for Image Net and 25 labels for each class for CIFAR... We sweep the learning rate from {0.01, 0.02, 0.05, 0.1, 0.2} and epochs from {30, 50}, choosing the hyperparameter based on a 12,000 images subset from the Image Net training set. We report the mean and standard deviation for 5 runs for (a) and (b) and 3 runs for (c). |
| Hardware Specification | Yes | For large-scale experiments, we train a Res Net-50 network using an unlabeled batch of size 4096 on 64 GPUs. For small-scale experiments, we train a WRN-28-2 network using an unlabeled batch of size 256 on 1 GPU. |
| Software Dependencies | No | The paper mentions using PyTorch implementations for baselines (e.g., Fix Match and Open Match), but it does not specify version numbers for any software dependencies (e.g., Python, PyTorch, CUDA) used for its own work. |
| Experiment Setup | Yes | Ro PAWS has three hyperparameters: the scale ratio of labeled and unlabeled batch r, the temperature for in-domain prior τprior, and the power of reweighted loss k. We set r = 5 for all experiments, and set τprior = 3.0 for Res Net-50 and τprior = 0.1 for WRN-28-2. We set k = 1 for all experiments except k = 3 for finetuning from Image Net pre-trained models. We use the LARS (You et al., 2017) optimizer with momentum value 0.9 and weight decay 10^-6. We linearly warm up the learning rate from 0.3 to 6.4 during the first 10 epochs and apply the cosine schedule afterward. |