Contrastive Learning is Spectral Clustering on Similarity Graph
Authors: Zhiquan Tan, Yifan Zhang, Jingqin Yang, Yang Yuan
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Motivated by our theoretical insights, we introduce the Kernel-Info NCE loss, incorporating mixtures of kernel functions that outperform the standard Gaussian kernel on several vision datasets. ... We propose a new Kernel-Info NCE loss with mixture of kernel functions that achieves better performance than the standard Gaussian kernel (Sim CLR) empirically on the benchmark vision datasets. ... In our experiments, we reproduce the baseline algorithm Sim CLR (Chen et al., 2020a), and replace Sim CLR s Gaussian kernel with other kernels. We then test against Sim CLR using Kernel-Info NCE loss on various benchmark vision datasets, including CIFAR-10/100 (Krizhevsky et al., 2009) and Tiny Image Net (Le & Yang, 2015). |
| Researcher Affiliation | Collaboration | 1Department of Mathematical Sciences, Tsinghua University 2IIIS, Tsinghua University 3Shanghai Artificial Intelligence Laboratory 4Shanghai Qizhi Institute {tanzq21, zhangyif21, yangjq21}@mails.tsinghua.edu.cn, yuanyang@tsinghua.edu.cn |
| Pseudocode | Yes | Algorithm 1 presents the pseudo-code for our empirical training procedure. ... Algorithm 2 Core loss function C |
| Open Source Code | Yes | For reproducibility, we share our code at https://github.com/yifanzhang-pro/ Kernel-Info NCE. |
| Open Datasets | Yes | In our experiments, we reproduce the baseline algorithm Sim CLR (Chen et al., 2020a), and replace Sim CLR s Gaussian kernel with other kernels. We then test against Sim CLR using Kernel-Info NCE loss on various benchmark vision datasets, including CIFAR-10/100 (Krizhevsky et al., 2009) and Tiny Image Net (Le & Yang, 2015). ... CIFAR-10 (Krizhevsky et al., 2009) and CIFAR-100 (Krizhevsky et al., 2009) are well-known classic image classification datasets. ... Tiny Image Net (Le & Yang, 2015) is a subset of Image Net (Deng et al., 2009). |
| Dataset Splits | Yes | Both CIFAR-10 and CIFAR-100 contain a total of 60k 32 32 labeled images of different classes, with 50k for training and 10k for testing. ... There are 200 different object classes in Tiny Image Net, with 500 training images, 50 validation images, and 50 test images for each class. |
| Hardware Specification | Yes | We train each encoder using the LARS optimizer (You et al., 2017), Lambda LR Scheduler in Py Torch, momentum 0.9, weight decay 10 6, batch size 256, and the aforementioned hyperparameters for 400 epochs on a single A-100 GPU. |
| Software Dependencies | Yes | We reproduce the Sim CLR algorithm using Py Torch Lightning (Team, 2022). ... The Py Torch Lightning Team. Pytorch-lightning: A machine learning library. https://github. com/Lightning-AI/lightning/releases/tag/1.8.6, December 2022. Version 1.8.6. |
| Experiment Setup | Yes | We employ Res Net50 (He et al., 2016) as the backbone and a 2-layer MLP (connected by a batch normalization (Ioffe & Szegedy, 2015) layer and a Re LU Nair & Hinton (2010) layer) with hidden dimensions 2048 and output dimensions 128 (or 256 in the concatenation kernel case). ... For each encoder training case, we randomly sample 500 hyperparameter groups (sample details are shown in Table 3). ... We train each encoder using the LARS optimizer (You et al., 2017), Lambda LR Scheduler in Py Torch, momentum 0.9, weight decay 10 6, batch size 256, and the aforementioned hyperparameters for 400 epochs on a single A-100 GPU. ... The linear head is trained using the SGD optimizer with a cosine learning rate scheduler, batch size 64, and weight decay 10 6 for 100 epochs. The learning rate starts at 0.3 and ends at 0. |