SIMPLE: Specialized Model-Sample Matching for Domain Generalization
Authors: Ziyue Li, Kan Ren, XINYANG JIANG, Yifei Shen, Haipeng Zhang, Dongsheng Li
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The experiments on Domain Bed show that our method achieves significant performance improvements (up to 12.2% for individual dataset and 3.9% on average) compared to state-of-the-art (SOTA) methods and further achieves 6.1% gain via enlarging the pretrained model pool. |
| Researcher Affiliation | Collaboration | Ziyue Li1 , Kan Ren2, Xinyang Jiang2, Yifei Shen2, Haipeng Zhang1, Dongsheng Li2 1Shanghai Tech University, 2Microsoft Research litzy0619owned@gmail.com, kan.ren@microsoft.com |
| Pseudocode | No | The paper describes methods in text and figures, but does not provide any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code and supplemental materials are available at https://seqml.github.io/simple. |
| Open Datasets | Yes | We conduct experiments on 5 real-world benchmark datasets including PACS (4 domains, 9,991 samples, 7 classes) (Li et al., 2017), VLCS (4 domains, 10,729 samples, 5 classes) (Fang et al., 2013), Office Home (4 domains, 15,588 samples, 65 classes) (Venkateswara et al., 2017), Terra Incognita (4 domains, 24,778 samples, 10 classes) (Beery et al., 2018), and Domain Net (6 domains, 586,575 samples, 345 classes) (Peng et al., 2019). |
| Dataset Splits | Yes | We use the training-domain validation set protocol for model selection. Specifically, one domain in a dataset is selected as the target domain and the rest as source domains, from which 20% of samples are used as the validation set. |
| Hardware Specification | Yes | To fairly compare training costs, we run experiments of ERM, SWAD, and SIMPLE on a single Nvidia Tesla V100 and compare their overall back-propagation time from the start of training to the end (or early-stop). |
| Software Dependencies | No | The paper mentions using Adam optimizer and Pytorch image models (via external links) but does not provide specific version numbers for these or other software dependencies. |
| Experiment Setup | Yes | Hyperparameter tuning. Here we state the details of hyperparameter tuning in our experiments. We use separate Adam optimizers (Zhang, 2018) for the ensemble network and label space adapter. Table 4 lists the hyperparameters to tune and their search space. |