Improving Contrastive Learning on Imbalanced Data via Open-World Sampling
Authors: Ziyu Jiang, Tianlong Chen, Ting Chen, Zhangyang Wang
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, using Image Net-100-LT (without labels) as the seed dataset and two noisy external data sources, we demonstrate that MAK can consistently improve both the overall representation quality and the class balancedness of the learned features, as evaluated via linear classifier evaluation on full-shot and few-shot settings. |
| Researcher Affiliation | Collaboration | Ziyu Jiang1, Tianlong Chen2, Ting Chen3, Zhangyang Wang2 1Texas A&M University, 2University of Texas at Austin, 3Google Research, Brain Team jiangziyu@tamu.edu, {tianlong.chen,atlaswang}@utexas.edu, iamtingchen@google.com |
| Pseudocode | Yes | Algorithm 1: A greedy heuristic to efficiently solve MAK. |
| Open Source Code | Yes | The code is available at: https://github.com/VITA-Group/MAK. |
| Open Datasets | Yes | We employ Image Net-100-LT as the seed training dataset, which is introduced in [11]. It is a long-tail version Image Net-100 [26]. ... Image Net-900 is composed by the rest part of Image Net [28] excluding Image Net100 [26]. ... the Oo D data is sampled from Places, a large scale scene-centric database. |
| Dataset Splits | No | The paper describes evaluation protocols and performance metrics on different subsets (e.g., 'balanced dataset', '1% samples of the full dataset') for fine-tuning and testing, but does not explicitly define a separate 'validation' split from its primary training data for hyperparameter tuning during the main model training phase. |
| Hardware Specification | Yes | Our codes are based on Pytorch [50], and all models are trained with NVIDIA A100 Tensor Core GPU. |
| Software Dependencies | No | Our codes are based on Pytorch [50] (no version number provided for Pytorch). |
| Experiment Setup | Yes | Training settings We conduct the experiments with the Sim CLR framework. We follow the settings of Sim CLR [1] including augmentations, projection head setting, optimizer, temperature, and learning rate. We use Resnet-50 [29] for all experiments (including feature extractor f). Sampling settings When calculating loss enforcing term. Augmentation repeat number M is set as 5 (see explanation at 3.4). For K-mean clustering of proximity, the number of centers K is set as 10. For the optimization process, the coefficient α is set as 0.3, and the candidate set size C is set as 1.5 K. As we will show in Section 3.3, the hyper-parameters choosing is not sensitive. ... B.1 Pre-training We identically follow [11] for pre-training settings except the epochs number: we pre-train for 1000 epochs for all our experiments following [1] (Including the feature extractor). B.2 Fine-tuning When fine-tuning for linear separability performance, the optimizer is set as SGD with momentum of 0.9 and initial learning rate of 30 following [51]. we train for 30 epochs and decrease the learning rate by 10 times at epochs 10 and 20. When fine-tuning for few-shot performance, we follow [2] fine-tuning from the first MLP projection layer. We train for 100 epochs with batch size 64. The initial lr is set as 0.02 and cosine learning rate decay without warm up is employed [2]. |