Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Domain Adaptation under Open Set Label Shift
Authors: Saurabh Garg, Sivaraman Balakrishnan, Zachary Lipton
NeurIPS 2022 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments across numerous semi-synthetic benchmarks on vision, language, and medical datasets demonstrate that our methods consistently outperform open set domain adaptation baselines, achieving 10 25% improvements in target domain accuracy. |
| Researcher Affiliation | Academia | Saurabh Garg, Sivaraman Balakrishnan, Zachary C. Lipton Machine Learning Department, Department of Statistics and Data Science, Carnegie Mellon University EMAIL |
| Pseudocode | Yes | Algorithm 1 Positive and Unlabeled learning post Label Shift Estimation (PULSE) framework |
| Open Source Code | Yes | Code is available at https://github.com/acmi-lab/Open-Set-Label-Shift |
| Open Datasets | Yes | For vision, we use CIFAR10, CIFAR100 [40] and Entity30 [61]. For language, we experiment with Newsgroups-20 (http://qwone.com/~jason/20Newsgroups/) dataset. Additionally, inspired by applications of OSLS in biology and medicine, we experiment with Tabula Muris [17] (Gene Ontology prediction), Dermnet (skin disease prediction https://dermnetnz.org/), and Break His [66] (tumor cell classification). |
| Dataset Splits | Yes | We repeat the same process on iid hold out data to obtain validation data with no target labels. |
| Hardware Specification | Yes | All experiments are run on an internal compute cluster with NVIDIA V100 GPUs. |
| Software Dependencies | No | The paper mentions using PyTorch for implementation but does not specify its version or any other software dependencies with version numbers (e.g., 'PyTorch (Paszke et al., 2019)' is a citation, not a version number). |
| Experiment Setup | Yes | We use default hyperparameters for all methods. For OSDA methods, we use default method specific hyperparameters introduced in their works. We provide precise details about hyperparameters, OSLS setup for each dataset and code in App. F.3. |