Unsupervised Visual Representation Learning via Mutual Information Regularized Assignment
Authors: Dong Hoon Lee, Sungik Choi, Hyunwoo J. Kim, Sae-Young Chung
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 4 Experiments In this section, we evaluate the representation quality learned via MIRA. We first provide the implementation details of our representation learning with MIRA (Sec. 4.1). We present our main results on linear, k-NN, semi-supervised learning, and transfer learning benchmarks in comparison to other self-supervised baselines (Sec. 4.2). Finally, we conduct an analysis of MIRA (Sec. 4.3). |
| Researcher Affiliation | Collaboration | Dong Hoon Lee KAIST donghoonlee@kaist.ac.kr Sungik Choi LG AI Research sungik.choi@lgresearch.ai Hyunwoo Kim LG AI Research hwkim@lgresearch.ai Sae-Young Chung KAIST sychungster@naver.com |
| Pseudocode | Yes | The pseudo-code of MIRA for representation learning with Eq. 8 is provided in the Appendix. |
| Open Source Code | Yes | Our implementation is available at https://github.com/movinghoon/mira. |
| Open Datasets | Yes | We train our model on the training set of the ILSVRC-2012 Image Net-1k dataset [18] without using class labels. We evaluate the representation learned by MIRA on the transfer learning benchmark with FGVC aircraft [41], Caltech-101 [25], Standford Cars [36], CIFAR-10/100 [37], DTD [15], Oxford 102 Flowers [42], Food-101 [5], Oxford-IIIT Pets [43], SUN397 [48], and Pascal VOC2007 [24] datasets. |
| Dataset Splits | Yes | We train a linear classifier on the top of the frozen trained backbone with the labeled training set of Image Net. We choose a learning rate with a local validation set in the Image Net train dataset and adjust the learning rate by cosine annealing schedule. We evaluate the representation quality by the linear classifier s performance on the validation set of Image Net. The final performance is evaluated on the model that is retrained on all training and validation sets with the found coefficient. |
| Hardware Specification | Yes | 8 TITAN V (from Table 8). 16 A100 (from Table 8). |
| Software Dependencies | No | The paper mentions using a 'LARS optimizer' and 'Res Net-50', but does not specify software versions for libraries like PyTorch, TensorFlow, or specific Python versions used for the experiments. |
| Experiment Setup | Yes | We use a batch size of 4096 and employ the LARS optimizer [52] with a weight decay of 10 6. We use linearly scaled learning rate of lr batch size/256 [27] with a base learning rate of 0.3. We adjust the learning rate with 10 epochs of a linear warmup followed by cosine scheduling. We also use an exponential moving average (EMA) network by default. When using EMA, we set the momentum update parameter to start from 0.99 and increase it to 1 by cosine scheduling. We use temperature scales of τs = 0.1, τt = 0.225 with a trade-off coefficient β = 2/3. We obtain soft pseudo-labels by 30 steps of the fixed point iteration. |