Self-Correcting Bayesian Optimization through Bayesian Active Learning
Authors: Carl Hvarfner, Erik Hellsten, Frank Hutter, Luigi Nardi
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section we showcase the performance of the SAL and SCore BO acquisition functions on a variety of tasks. For active learning, SAL shows state-of-the-art performance on a majority of benchmarks, and is more robust than the baselines. For the optimization tasks, SCore BO more efficiently learns the model hyperparameters, and outperforms prominent Bayesian optimization acquisition functions on a variety of tasks. All experiments are implemented in Bo Torch [2]1. |
| Researcher Affiliation | Collaboration | Carl Hvarfner carl.hvarfner@cs.lth.se Lund University Erik Orm Hellsten erik.hellsten@cs.lth.se Lund University Frank Hutter fh@cs.uni-freiburg.de University of Freiburg Luigi Nardi luigi.nardi@cs.lth.se Lund University Stanford University DBtune |
| Pseudocode | Yes | Algorithm 1 SCore BO iteration |
| Open Source Code | Yes | The complete experimental setup is presented in detail in Appendix B, and our code is publicly available at https: //github.com/hvarfner/scorebo.git. |
| Open Datasets | Yes | To evaluate the performance of SAL, we compare it with BALD, BQBC and QBMGP on the same six functions used by Riis et al. [50]: Gramacy (1D) has a periodicity that is hard to distinguish from noise, Higdon and Gramacy (2D) varies in characteristics in different regions, whereas Branin, Hartmann-6 and Ishigami have a generally nonlinear structure. |
| Dataset Splits | Yes | We evaluate each method on their predictive power, measured by the negative Marginal Log Likelihood (MLL) of the model predictions over a large set of validation points. |
| Hardware Specification | Yes | All experiments are carried out on Intel Xeon Gold 6130 CPUs. |
| Software Dependencies | Yes | All experiments are implemented in Bo Torch [2]1. ... https://botorch.org/ (v0.8.4) and For both the active learning and BO experiments, we run NUTS [25] in Pyro [5] to draw samples from the GP posterior over hyperparameters. |
| Experiment Setup | Yes | The complete experimental setup is presented in detail in Appendix B... Tab. 3 displays the parameters of the MCMC in detail, as well as other relevant parameters of various MC estimations throughout the article. For the active learning experiments, we mimic the experimental setup used in Riis et al. [50], and put a log-normal distribution LNp0, 3q on the lengthscales, outputscale variance and noise variance. Furthermore, we consider the mean constant c as a learnable parameter in the BO experiments, with a conventional Np0, 1q prior on the standardized inputs. |