Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift
Authors: Yihao Xue, Siddharth Joshi, Dang Nguyen, Baharan Mirzasoleiman
ICLR 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We further support our theoretical findings with experiments, including a well-designed synthetic experiment and experiments on real datasets, including MSCOCO, Conceptual Captions, and shifted versions of Image Net. |
| Researcher Affiliation | Academia | Yihao Xue, Siddharth Joshi, Dang Nguyen, Baharan Mirzasoleiman Department of Computer Science, University of California, Los Angeles EMAIL, EMAIL, EMAIL, EMAIL |
| Pseudocode | No | The paper describes mathematical formulations and processes but does not include explicit pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide an explicit statement or link for the release of open-source code for the described methodology. |
| Open Datasets | Yes | We validate our theoretical findings through experiments, including a well-designed synthetic experiment and an experiment involving training CLIP models on MSCOCO (Lin et al., 2014)/Conceptual Captions (Sharma et al., 2018) and evaluating them on shifted Image Nets. |
| Dataset Splits | Yes | The dataset is divided into Training, Validation, and Test splits. The Training split includes 3,318,333 pairs in which a subset of 2,007,528 has machine-generated labels Ng et al. (2020). ... To create the training and validation datasets, we split the subset with the 7:3 ratio in a stratified fashion. |
| Hardware Specification | Yes | Each experiment is run on 1 NVIDIA A6000. |
| Software Dependencies | No | The paper mentions using 'Pytorch' but does not specify its version number or any other software dependencies with specific version numbers. |
| Experiment Setup | Yes | We use momentum SGD as the optimizer with a learning rate of 0.01, weight decay of 0.001, momentum of 0.9, a batch size of 128. The model is trained for 100 epochs. |