Provable Guarantees for Gradient-Based Meta-Learning
Authors: Maria-Florina Balcan, Mikhail Khodak, Ameet Talwalkar
ICML 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We use experiments in both convex and deep learning settings to verify and demonstrate the applicability of our theory. We verify several assumptions and implications of our theory using a new meta-learning dataset we introduce consisting of text-classification tasks solvable using convex methods. We further study the empirical suggestions of our theory in the deep learning setting. |
| Researcher Affiliation | Collaboration | 1Carnegie Mellon University 2Determined AI. |
| Pseudocode | Yes | Algorithm 1: The generic online-within-online algorithm we study. Algorithm 2: Follow-the-Meta-Regularized-Leader (Ephemeral) meta-algorithm for meta-learning. |
| Open Source Code | No | The paper mentions a new dataset is available in the supplement but does not provide a concrete access link or explicit statement about the release of source code for their methodology. |
| Open Datasets | Yes | We introduce a new dataset of 812 classification tasks, each consisting of sentences from one of four Wikipedia pages which we use as labels. It is derived from the raw super-set of the Wiki3029 corpus collected by Arora et al. (2019). We call the new dataset Mini-Wiki and make it available in the supplement. We study modifications of Reptile on 5-way and 20-way Omniglot (Lake et al., 2017) and 5-way Mini-Image Net classification (Ravi & Larochelle, 2017) using the same networks as Nichol et al. (2018). |
| Dataset Splits | No | The paper mentions 'meta-train time' and 'meta-test time' but does not provide specific details on validation splits (percentages, sample counts, or explicit mention of a validation set). |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU/CPU models, processor types, or memory amounts used for running its experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details with version numbers (e.g., library or solver names with version numbers) needed to replicate the experiment. |
| Experiment Setup | Yes | For Ephemeral we use the FAL variant with OGD as the withintask algorithm, with learning rate set using the average deviation of the task parameters from the mean parameter, as suggested in Remark 2.1. For MAML, we use grid search to determine the within-task and meta-update learning rates. The Reptile authors do indeed use more than 5 task samples 10 for Omniglot and 15 for Mini-Image Net. Similarly, they use far fewer within-task gradient steps 5 for Omniglot and 8 for Mini-Image Net at meta-train time than the 50 iterations used for evaluation. |