Large-Margin Contrastive Learning with Distance Polarization Regularizer
Authors: Shuo Chen, Gang Niu, Chen Gong, Jun Li, Jian Yang, Masashi Sugiyama
ICML 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | empirically, the superiority of LMCL is demonstrated across multiple domains, i.e., image classification, sentence representation, and reinforcement learning. We conduct extensive experiments on synthesis and real-world datasets to validate the superiority of our method over the state-of-the-art CL approaches. |
| Researcher Affiliation | Academia | 1RIKEN Center for Advanced Intelligence Project, Japan; 2PCA-Lab, School of Computer Science and Engineering, Nanjing University of Science and Technology, China; 3Graduate School of Frontier Sciences, The University of Tokyo, Japan. |
| Pseudocode | Yes | Algorithm 1 Solving Eq. (9) via Adam. |
| Open Source Code | No | Not found. |
| Open Datasets | Yes | Specifically, here we choose the popular method Sim CLR (Chen et al., 2020a) as our framework to learn the embedding ϕ on CIFAR-10 (Krizhevsky et al., 2009) dataset using the Adam optimizer (Reddi et al., 2018). We also compare our method with three additional state-of-the-art methods including debiased contrastive learning (DCL) (Chuang et al., 2020), hard negative based contrastive learning (HCL) (Robinson et al., 2020), and the clustering based method (Sw AV) (Caron et al., 2020) on STL-10 (Coates et al., 2011), CIFAR-10 (Krizhevsky et al., 2009), and Image Net-100 (Russakovsky et al., 2015) datasets. In this subsection, we employ the Book Corpus dataset (Kiros et al., 2015) to evaluate the performance of all compared methods on six text classification tasks... Here the contrastive unsupervised representations for reinforcement learning (CURL) (Laskin et al., 2020) method is employed to perform imagebased policy control on representation learned by the CL algorithm. All methods are tested on the Deep Mind control suite (Tassa et al., 2018), which consists of six control tasks listed in Tab. 5. |
| Dataset Splits | Yes | Here the 10-fold cross validation is adopted, and the average classification accuracy is listed in Tab. 4. |
| Hardware Specification | No | Not found. |
| Software Dependencies | No | Specifically, here we choose the popular method Sim CLR (Chen et al., 2020a) as our framework to learn the embedding ϕ on CIFAR-10 (Krizhevsky et al., 2009) dataset using the Adam optimizer (Reddi et al., 2018). All methods are fairly implemented by the Res Net50 with the same training epoch 100. |
| Experiment Setup | Yes | For these positive pairs and negative pairs, we use the Adam optimizer (learning rate = 0.001) for both the conventional CL (i.e., Eq. (1)) and our proposed LMCL (i.e., Eq. (9) with λ=0.1). All methods are fairly implemented by the Res Net50 with the same training epoch 100. The regularization parameter λ of our method is fixed to 0.1. The thresholds δ+ and δ are fixed to 0.1 and 0.5, respectively. |