How Tight Can PAC-Bayes be in the Small Data Regime?
Authors: Andrew Foong, Wessel Bruinsma, David Burt, Richard Turner
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We study synthetic one-dimensional classification tasks in which it is feasible to meta-learn both the prior and the form of the bound to numerically optimise for the tightest bounds possible. |
| Researcher Affiliation | Collaboration | Andrew Y. K. Foong University of Cambridge ykf21@cam.ac.uk Wessel P. Bruinsma University of Cambridge Invenia Labs wpb23@cam.ac.uk David R. Burt University of Cambridge drb62@cam.ac.uk Richard E. Turner University of Cambridge ret26@cam.ac.uk |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code to reproduce all experiments can be found at https://github.com/cambridge-mlg/pac-bayes-tightness-small-data. |
| Open Datasets | No | We choose especially simple learning tasks synthetic 1-dimensional binary classification problems, generated by thresholding Gaussian process (GP) samples. |
| Dataset Splits | Yes | Test set bounds rely on a subset of data which is not used to select the hypothesis, called a test set or held-out set. Let S = Strain Stest, with |S| = N, |Strain| = Ntrain and |Stest| = Ntest. [...] To compare PAC-Bayes DDPs against test set bounds, we sweep the prior/train set proportion from 0 to 0.8 and see what the tightest value obtained is. |
| Hardware Specification | No | The paper does not provide specific hardware details (like GPU/CPU models or memory amounts) used for running its experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details with version numbers, such as libraries or frameworks used in the experiments. |
| Experiment Setup | Yes | Hyperparameter details are given in Appendix I.7. |