Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
The Close Relationship Between Contrastive Learning and Meta-Learning
Authors: Renkun Ni, Manli Shu, Hossein Souri, Micah Goldblum, Tom Goldstein
ICLR 2022 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct self-supervised training on both the CIFAR-10 and Image Net datasets (Krizhevsky et al., 2009; Deng et al., 2009). Following Chen et al. (2020a), we evaluate pre-trained representations in a linear evaluation setting, where feature extractors are frozen, and a classification head is stacked on top and tuned. In addition, we test the performance of the pre-trained feature extractors on downstream tasks such as transfer learning and semi-supervised learning with 1% and 10% of labeled data. Evaluation and dataset details can be found in Appendix A.2. |
| Researcher Affiliation | Academia | Renkun Ni University of Maryland EMAIL Manli Shu University of Maryland EMAIL Hossein Souri Johns Hopkins University EMAIL Micah Goldblum University of Maryland EMAIL Tom Goldstein University of Maryland EMAIL |
| Pseudocode | Yes | Algorithm 1: Meta-Learning Framework for Self-Supervised Learning |
| Open Source Code | Yes | Our Pytorch implementation can be found on: https://github.com/Renkun Ni/Meta Contrastive |
| Open Datasets | Yes | We conduct self-supervised training on both the CIFAR-10 and Image Net datasets (Krizhevsky et al., 2009; Deng et al., 2009). |
| Dataset Splits | Yes | For each dataset, we use the backbone (Res Net-50) pre-trained on Image Net as an initialization for the feature extractor of the downstream classification model. In contrast to linear evaluation, we fine-tune the entire model on the given dataset for 20,000 iterations with the best hyperparameter setting selected on its validation split. Details of our hyperparameter selection are included in Appendix A.2. All models are pre-trained on Image Net for 100 epochs. |
| Hardware Specification | No | We train the model on CIFAR-10 with the LARS optimizer (You et al., 2019) and batch size 1024 for 1000 epochs (with 4 GPUs). On Image Net, we use the same optimizer and batch size of 256, and we train for 100 epochs (with 8 GPUs). This does not provide specific hardware models. |
| Software Dependencies | No | Our Pytorch implementation can be found on: https://github.com/Renkun Ni/Meta Contrastive This only mentions the framework, not specific versions. |
| Experiment Setup | Yes | We use a Res Net-18 backbone for all experiments on CIFAR-10 and Res Net50 for those on Image Net. We train the model on CIFAR-10 with the LARS optimizer (You et al., 2019) and batch size 1024 for 1000 epochs (with 4 GPUs). On Image Net, we use the same optimizer and batch size of 256, and we train for 100 epochs (with 8 GPUs). For Image Net pre-training, we follow the hyperparameter setting in Chen et al. (2020a), including baseline data augmentation methods, dimension of the latent space, and learning rate decay schedule. For CIFAR-10 pre-training, we use the same CIFAR-10 specific hyperparameters as Sim CLR again. For BYOL, we use the same learning rate schedule as meta-learners and start with learning rate 4. In addition, both the projector and predictor in BYOL are two-layer MLPs with hidden dimension 2048 and output dimension 256. More details can be found in Appendix A.1. |