Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stability
Authors: Alec Farid, Anirudha Majumdar
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show that our bound provides a tighter guarantee than other bounds on a toy non-convex problem on the unit sphere and a text-based classification example. We also present a practical regularization scheme motivated by the bound in settings where the bound is loose and demonstrate improved performance over baseline techniques. |
| Researcher Affiliation | Academia | Alec Farid Anirudha Majumdar Department of Mechanical and Aerospace Engineering, Princeton University {afarid, ani.majumdar}@princeton.edu |
| Pseudocode | Yes | Algorithm 1 PAC-BUS: meta-learning via PAC-Bayes and Uniform Stability |
| Open Source Code | Yes | All the code required to run the following examples is available at https://github.com/irom-lab/PAC-BUS. |
| Open Datasets | Yes | Omniglot [35], Mini-Wiki benchmark introduced in [34]. This is derived from the Wiki3029 dataset presented in [9]. |
| Dataset Splits | Yes | For all methods, 10,000 tasks are sampled as meta-training data and 1,000 tasks are sampled as held-out meta-test data. For the prior training step, all methods use 100 meta-training tasks. |
| Hardware Specification | No | It took multiple weeks of computation time on Amazon Web Services (AWS) instances to train and compute all networks and results we present in this paper. However, no specific instance types or hardware specifications (e.g., GPU/CPU models) are provided. |
| Software Dependencies | Yes | We use Python 3.8.3, learn2learn v0.1.0, PyTorch 1.5.0, torch-imle 0.0.1, and scipy 1.4.1. |
| Experiment Setup | Yes | We choose the softmax-activated cross-entropy loss, CELs, as the loss function. (from Section 5.1). Additionally, the Appendix A.11 provides tables of hyperparameters (e.g., 'Table 5: Hyperparameters for experiments on the classification on ball problem.' listing learning rates, epochs, etc.) |