Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning
Authors: Namyeong Kwon, Hwidong Na, Gabriel Huang, Simon Lacoste-Julien
ICLR 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We train MAML on mini Image Net (Vinyals et al., 2016) training split; we then apply our method on the resulting checkpoint. We evaluate our model on the test split of mini Image Net for the samedomain setting as well as CUB-200-2011 (Welinder et al., 2010), Traffic Sign (Houben et al., 2013) and VGG Flower (Nilsback & Zisserman, 2008) for the cross-domain setting. These datasets are denoted as Mini, Birds, Signs and Flowers respectively. A desirable feature for an optimizer is to maintain good performance in a broad range of stepsizes (Asi & Duchi, 2019). Therefore, we evaluate our approach not only at the optimal stepsize, but also over a broad range of base stepsizes2 from 10 4 to 1. We evaluate the performance with three metrics: All, Top-1 and Top-40%.3 If two methods have comparable Top-1 performance, but one has better Top-40% performance, then that method is more robust to the choice of base stepsize. Detailed experimental setup and pretrained model selection are included in Appendix A.3. |
| Researcher Affiliation | Collaboration | Namyeong Kwon, Hwidong Na Samsung Advanced Institute of Technology (SAIT), South Korea {ny.kwon,hwidong.na}@samsung.com Gabriel Huang Mila, Universit e de Montr eal Simon Lacoste-Julien Mila, Universit e de Montr eal SAIT AI Lab, Montreal |
| Pseudocode | Yes | Algorithm 1: Uncertainty-based Gradient Steps at Test-time |
| Open Source Code | Yes | Our code is available at https://github.com/Namyeong K/USA_UFGSM/. |
| Open Datasets | Yes | We train MAML on mini Image Net (Vinyals et al., 2016) training split; we then apply our method on the resulting checkpoint. We evaluate our model on the test split of mini Image Net for the samedomain setting as well as CUB-200-2011 (Welinder et al., 2010), Traffic Sign (Houben et al., 2013) and VGG Flower (Nilsback & Zisserman, 2008) for the cross-domain setting. These datasets are denoted as Mini, Birds, Signs and Flowers respectively. |
| Dataset Splits | Yes | A pretrained model was selected with the validation accuracy among mini Image Net training checkpoints. We used the checkpoint with the highest validation accuracy as the pretrained model θ0. The highest test accuracy we achieved was 49.24% during meta-training. For the checkpoint that we chose with the highest validation performance, the test accuracy was 47.58%. |
| Hardware Specification | No | The paper does not specify the hardware used for experiments, such as specific CPU or GPU models. |
| Software Dependencies | No | The paper mentions using SGD and Adam optimizers, but does not provide specific version numbers for any software libraries, frameworks, or programming languages used (e.g., PyTorch version, Python version, etc.). |
| Experiment Setup | Yes | Our baseline model was trained using the same hyperparameters with mini Image Net training of MAML except the inner loop stepsize. The inner loop stepsize was set to 0.1 for reproducing the 5-way 1-shot accuracy reported in the original MAML paper. We trained the model for 150,000 iterations. A pretrained model was selected with the validation accuracy among mini Image Net training checkpoints. The size of ensemble is M = 5 which is the same as the deep ensembles (Lakshminarayanan et al., 2017). The parameter for the FGSM is ϵ = 0.05. The scale value a of Gaussian random perturbation for ensemble model training is σ = 0.05. The gradient step is T = 10 which is same with mini Image Net test of MAML. |