Automated Model Selection with Bayesian Quadrature
Authors: Henry Chai, Jean-Francois Ton, Michael A. Osborne, Roman Garnett
ICML 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our method produces more-accurate posterior estimates using fewer likelihood evaluations than standard Bayesian quadrature and Monte Carlo estimators, as we demonstrate on synthetic and real-world examples. |
| Researcher Affiliation | Collaboration | 1Department of Computer Science and Engineering, Washington University in St. Louis, Saint Louis, MO, USA 2Department of Statistics, University of Oxford, Oxford, United Kingdom 3Department of Engineering Science, University of Oxford, Oxford, United Kingdom 4Mind Foundry, Oxford, United Kingdom. |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any explicit statements or links indicating that source code for the methodology is available. |
| Open Datasets | Yes | For our synthetic experiments, we consider a model selection task between two zero-mean GP models...The observed dataset D consists of 5d observations from a d-dimensional, zero-mean GP with a squared exponential covariance...For both of the experiments below, the diffeomorphism associated with our implementation of reversible jump MCMC is again the identity function and the corresponding Jacobian factor is 1. The intractable integrals associated with our proposed method are estimated using quasi-Monte Carlo (Caflisch, 1998). We evaluated all methods on the absolute error of their estimates of the log Bayes factor: log Bij = log zi log zj (Jeffreys, 1961; Kass & Raftery, 1995), another potential quantity of interest in model selection tasks. We consider the absolute error instead of the fractional error as the target quantity is a log value. We make use of Bartolucci et al. (2006) s work to translate the output Markov chain into a log odds estimate. Our real-world application is a model selection problem from the field of astrophysics... We select 20 spectra from phase III of the Sloan Digital Sky Survey (SDSS III) (Eisenstein et al., 2011) |
| Dataset Splits | No | The paper describes its data acquisition strategy (e.g., 'randomly sampled likelihood observations', 'allot a budget') for active learning, but it does not specify explicit train/validation/test dataset splits with percentages or counts for static datasets. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., CPU/GPU models, memory) used for running the experiments. It only vaguely mentions 'computational resources'. |
| Software Dependencies | No | The paper does not provide specific version numbers for software dependencies or libraries used (e.g., Python, PyTorch, CUDA versions). |
| Experiment Setup | Yes | For all BQ methods, constant-mean GP priors with Mat ern covariance functions (ν = 3/2) were placed on the log of the model likelihoods and all GP hyperparameters were fit in accordance with the framework defined by Chai & Garnett (2019). Our implementation of round-robin BQ uses uncertainty sampling to select locations to observe log-likelihoods, as proposed by Gunter et al. (2014). We allot a budget of 50d total likelihood evaluations and initialize each BQ based method with 5d randomly sampled likelihood observations from both model parameter spaces. |