Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization
Authors: Alexandre Rame, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Leon Bottou, David Lopez-Paz
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, it improves the state of the art on the reference Domain Bed benchmark for out-of-distribution generalization. We show the efficacy of model ratatouille in Section 4, where we set a new state of the art on Domain Bed (Gulrajani & Lopez-Paz, 2021), the reference benchmark evaluating OOD generalization. |
| Researcher Affiliation | Collaboration | 1Meta AI, Paris, France 2Sorbonne Universit e, CNRS, ISIR, Paris, France 3NYU, New-York, USA 4Valeo.ai, Paris, France. Correspondence to: Alexandre Ram e <alexandre.rame@isir.upmc.fr>. |
| Pseudocode | No | Ratatouille follows this five-step recipe. 1. Download a featurizer ϕpt pre-trained on task T0. 2. Fine-tune ϕpt on each auxiliary task Ti, obtaining (waux i , ϕaux i ) for i = 0, . . . , M 1. 3. Replace each waux i by wlp, obtained by linear probing the original pre-trained model ϕpt on the target task T. 4. Fine-tune each (wlp, ϕaux i ) on the target task T, obtaining θi = (wi, ϕi) for i = 0, . . . , M 1. 5. Return as final model PM 1 i=0 λi θi. |
| Open Source Code | Yes | Our code is released here. Our code is released at https://github.com/facebookresearch/Model Ratatouille. |
| Open Datasets | Yes | Domain Bed contains five real-world datasets: PACS (Li et al., 2017), VLCS (Fang et al., 2013), Office Home (Venkateswara et al., 2017), Terra Incognita (Beery et al., 2018) and Domain Net (Peng et al., 2019). |
| Dataset Splits | Yes | Domains are split into 80% (used as training and evaluation) and 20% (used as validation). We consider the training-domain validation set protocol. From each run, we thus take the weights at the epoch with maximum accuracy on the ID validation dataset. |
| Hardware Specification | No | This paradigm could also leverage the utilization of volunteer computing with single-GPU desktop machines, and complement approaches like Learning@home (Ryabinin & Gusev, 2020) or Petals (Borzunov et al., 2022). |
| Software Dependencies | No | The network is a Res Net-50 (He et al., 2016) pretrained on Image Net (Russakovsky et al., 2015). The optimizer is Adam (Kingma & Ba, 2015). |
| Experiment Setup | Yes | Table 3. Hyperparameters, their default values and distributions for random search. For each dataset, we perform a random search of 20 trials on the mild hyperparameter distributions described in Table 3. We use a Res Net-50 (He et al., 2016) pre-trained on Image Net, with a dropout layer before the newly added dense layer and fine-tuned with frozen batch normalization layers. The optimizer is Adam (Kingma & Ba, 2015). ... All runs are trained for 5k steps, except on Domain Net for 15k steps... validation accuracy is calculated every 50 steps for VLCS, 500 steps for Domain Net and 100 steps for others. |