Robust Contrastive Learning Using Negative Samples with Diminished Semantics
Authors: Songwei Ge, Shlok Mishra, Chun-Liang Li, Haohan Wang, David Jacobs
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this paper, we show that by generating carefully designed negative samples, contrastive learning can learn more robust representations with less dependence on such features. Contrastive learning utilizes positive pairs that preserve semantic information while perturbing superficial features in the training images. Similarly, we propose to generate negative samples in a reversed way, where only the superfluous instead of the semantic features are preserved. We develop two methods, texture-based and patch-based augmentations, to generate negative samples. These samples achieve better generalization, especially under out-of-domain settings. |
| Researcher Affiliation | Collaboration | Songwei Ge Univeristy of Maryland songweig@cs.umd.edu Shlok Mishra Univeristy of Maryland shlokm@cs.umd.edu Haohan Wang Carnegie Mellon University haohanw@cs.cmu.edu Chun-Liang Li Google Cloud AI chunliang@google.com David Jacobs Univeristy of Maryland dwj@cs.umd.edu |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our code, trained models, and Image Net-Texture dataset can be found at https://github.com/ Songwei Ge/Contrastive-Learning-with-Non-Semantic-Negatives. |
| Open Datasets | Yes | Our code, trained models, and Image Net-Texture dataset can be found at https://github.com/ Songwei Ge/Contrastive-Learning-with-Non-Semantic-Negatives. We evaluate our methods with two contrastive learning methods, Mo Co [18, 8] and BYOL [16], on three datasets, Image Net-100, Image Net-1K and STL10. |
| Dataset Splits | Yes | We follow the hyperparameters used in [31] to train Mo Co-v2 on the Image Net-100 dataset with a memory bank size k = 16384 or a halved memory bank size. We also conduct experiments following the hyperparameters in a concurrent study [44] except that we keep k = 16384 for our method... We repeat the experiments, including both the pretraining and linear evaluation, for 3 runs and report the mean and standard deviation in Table 1. |
| Hardware Specification | No | The paper mentions 'multi-threaded CPU support' for texture synthesis but does not provide specific details such as CPU model, GPU models, or any other hardware specifications used for running experiments. |
| Software Dependencies | No | The paper mentions 'open-source software built on these methods [12, 52, 1] with multi-threaded CPU support implemented in Rust' and provides a link to 'https://github.com/Embark Studios/texture-synthesis', but it does not specify version numbers for Rust or any other key software components. |
| Experiment Setup | Yes | We follow the hyperparameters used in [31] to train Mo Co-v2 on the Image Net-100 dataset with a memory bank size k = 16384 or a halved memory bank size. We also conduct experiments following the hyperparameters in a concurrent study [44] except that we keep k = 16384 for our method. For patch-based augmentation parameters, we use patch size sampled from a uniform distribution d U(16, 72). The parameter α is indicated behind each model name. |