Iterative Feature Matching: Toward Provable Domain Generalization with Logarithmic Environments
Authors: Yining Chen, Elan Rosenfeld, Mark Sellke, Tengyu Ma, Andrej Risteski
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We perform experiments on a Gaussian dataset and a semi-synthetic Noised MNIST dataset [23]. Our results in Section 6 suggest that practitioners may benefit from feature matching algorithms when the distinguishing property of the signal feature is indeed conditional distributional invariance, and may get additional advantage via matching at multiple layers with diminishing dimensions, echoing existing empirical observations [27, 29]. Figures 1 and 2 show that IFM and CORAL have much smaller environment complexity compared to ERM and IRM in both datasets. |
| Researcher Affiliation | Academia | Yining Chen Stanford University cynnjjs@stanford.edu Elan Rosenfeld Carnegie Mellon University elan@cmu.edu Mark Sellke Stanford University msellke@stanford.edu Tengyu Ma Stanford University tengyuma@stanford.edu Andrej Risteski Carnegie Mellon University aristesk@andrew.cmu.edu |
| Pseudocode | Yes | Algorithm 1 Iterative Feature Matching (IFM) algorithm |
| Open Source Code | Yes | 3. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Code attached in supplemental file. |
| Open Datasets | Yes | Noised MNIST is a 10-way semi-synthetic classification task modified from Le Cun and Cortes [23]. (Section 6). The reference [23] states: Y. Le Cun and C. Cortes. MNIST handwritten digit database. 2010. URL http://yann.lecun. com/exdb/mnist/. |
| Dataset Splits | No | In domain generalization, we are given a set of E training environments Etr indexed by e [E],1 and a set of test environments Ets. The paper describes splitting environments into training and test, and within Algorithm 1, it mentions 'Uniformly randomly sample Et training environments without replacement.' This refers to splitting environments for iterative matching, not typical train/validation/test data splits. No mention of validation data split. |
| Hardware Specification | Yes | All experiments were run on a server with an NVIDIA RTX 3090 GPU. (Appendix C) |
| Software Dependencies | No | The paper mentions using a 'server with an NVIDIA RTX 3090 GPU' for experiments but does not provide specific software dependencies with version numbers (e.g., Python, PyTorch, CUDA versions). |
| Experiment Setup | Yes | All hyperparameters were tuned on a small validation set (separate from test environments) and remained constant across all methods. For the Gaussian dataset, we use linear predictors and train all models for 100 epochs using Adam optimizer with learning rates of 1e-3. Batch size is 128. For Noised MNIST, we train for 100 epochs using Adam optimizer with learning rates of 1e-3. Batch size is 256. For CORAL+ON and IFM, we use λ1 = 1, λ2 = 1e-3 and decay learning rate by 0.1 every 50 epochs. (Appendix C) |