A theoretical study of inductive biases in contrastive learning
Authors: Jeff Z. HaoChen, Tengyu Ma
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this work, we provide the first theoretical analysis of self-supervised learning that incorporates the effect of inductive biases originating from the model class. We instantiate our theory on several synthetic data distributions, and provide empirical evidence to support the theory. We provide experimental results to support our theory. |
| Researcher Affiliation | Academia | Jeff Z. Hao Chen & Tengyu Ma Department of Computer Science Stanford University {jhaochen,tengyuma}@stanford.edu |
| Pseudocode | No | The paper contains mathematical derivations, theorems, and definitions, but it does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not contain any statement about releasing source code or provide links to a code repository. |
| Open Datasets | Yes | We run this test with a ResNet-18 model on CIFAR-10 and compute the br for r {10, 100, 500} list the results the table below. |
| Dataset Splits | No | The paper mentions 'CIFAR-10 training set' and 'test set' but does not specify the splits (e.g., percentages, sample counts) for training, validation, or testing. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for experiments, such as CPU or GPU models, or memory specifications. |
| Software Dependencies | No | The paper describes the models and training process but does not list specific software dependencies with version numbers (e.g., Python, PyTorch, CUDA versions). |
| Experiment Setup | Yes | We train with SGD using initial learning rate 0.01 and decays with cosine schedule. All experiments are run for 200 epochs. We test with r {10, 100, 500} and grid search using λ {0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000}, the result for each configurate is listed in the table below. |