Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations
Authors: Jeff Z. HaoChen, Colin Wei, Ananya Kumar, Tengyu Ma
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 5 Simulations We empirically show that our proposed Algorithm 1 achieves good performance on the unsupervised domain adaptation problem. We conduct experiments on BREEDS [Santurkar et al., 2020] a dataset for evaluating unsupervised domain adaptation algorithms (where the source and target domains are constructed from Image Net images). For pre-training, we run the spectral contrastive learning algorithm [Hao Chen et al., 2021] on the joint set of source and target domain data. Unlike the previous convention of discarding the projection head, we use the output after projection MLP as representations, because we find that it significantly improves the performance (for models learned by spectral contrastive loss) and is more consistent with the theoretical formulation. Given the pre-trained representations, we run Algorithm 1 with different choices of t. For comparison, we use the linear probing baseline where we train a linear head with logistic regression on the source domain. The table below lists the test accuracy on the target domain for Living-17 and Entity-30 two datasets constructed by BREEDS. Additional details can be found in Section A. |
| Researcher Affiliation | Academia | Jeff Z. Hao Chen Stanford University Colin Wei Stanford University Ananya Kumar Stanford University Tengyu Ma Stanford University |
| Pseudocode | Yes | Algorithm 1 Preconditioned feature averaging (PFA) |
| Open Source Code | No | Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] |
| Open Datasets | Yes | We conduct experiments on BREEDS [Santurkar et al., 2020] a dataset for evaluating unsupervised domain adaptation algorithms (where the source and target domains are constructed from Image Net images). ... The table below lists the test accuracy on the target domain for Living-17 and Entity-30 two datasets constructed by BREEDS. |
| Dataset Splits | No | The table below lists the test accuracy on the target domain for Living-17 and Entity-30 two datasets constructed by BREEDS. Additional details can be found in Section A. |
| Hardware Specification | No | Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] |
| Software Dependencies | No | For pre-training, we run the spectral contrastive learning algorithm [Hao Chen et al., 2021] on the joint set of source and target domain data. |
| Experiment Setup | No | For pre-training, we run the spectral contrastive learning algorithm [Hao Chen et al., 2021] on the joint set of source and target domain data. ... Given the pre-trained representations, we run Algorithm 1 with different choices of t. For comparison, we use the linear probing baseline where we train a linear head with logistic regression on the source domain. The table below lists the test accuracy on the target domain for Living-17 and Entity-30 two datasets constructed by BREEDS. Additional details can be found in Section A. |