Model-Aware Contrastive Learning: Towards Escaping the Dilemmas
Authors: Zizheng Huang, Haoxing Chen, Ziqi Wen, Chao Zhang, Huaxiong Li, Bo Wang, Chunlin Chen
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive remarkable empirical results in vision, sentence, and graph modality validate our approach s general improvement for representation learning and downstream tasks. |
| Researcher Affiliation | Collaboration | 1Nanjing University 2Ant Group 3China Telecom. |
| Pseudocode | Yes | A simple example pseudocode of Eqn.(11) is shown as Algorithm 1. |
| Open Source Code | No | The paper states 'Codes of models are implemented on mmselfsup (Contributors, 2021)' which refers to a third-party toolbox, but does not provide an explicit statement or link for the open-source code of their own described methodology (MACL). |
| Open Datasets | Yes | We mainly experiment on the Image Net ILSVRC-2012 (i.e., Image Net-1K) (Deng et al., 2009) and use standard Res Net-50 (He et al., 2016) as image encoders. CIFAR10 (Krizhevsky et al., 2009) and the subset Image Net-100 (Tian et al., 2020a) are also considered. ... We conduct experiments on six commonly used benchmarks (Morris et al., 2020). |
| Dataset Splits | Yes | We mainly experiment on the Image Net ILSVRC-2012 (i.e., Image Net-1K) (Deng et al., 2009) and use standard Res Net-50 (He et al., 2016) as image encoders. CIFAR10 (Krizhevsky et al., 2009) and the subset Image Net-100 (Tian et al., 2020a) are also considered. ... For linear evaluation, the trained CL models are evaluated by fine-tuning a linear classifier for 100 epochs with 128-batch size on top of frozen backbones. |
| Hardware Specification | Yes | Codes of models are implemented on mmselfsup (Contributors, 2021) with several Tesla A100 80G GPUs. |
| Software Dependencies | Yes | algorithms are performed based on Huggingface s transformers package1 and evaluated with Sent Eval toolkit2. ... 1https://github.com/huggingface/transformers,version 4.2.1. |
| Experiment Setup | Yes | Models optimizations are completed by LARS with a base learning rate of 0.3 (0.3 Batch Size/256) and weight decay of 1e-6. We also use the cosine decay learning rate schedule with 10 epochs warmup. Parameters {τ0, α, A0} are set to {0.1, 0.5, 0}. |