Efficient Nonmyopic Bayesian Optimization via One-Shot Multi-Step Trees
Authors: Shali Jiang, Daniel Jiang, Maximilian Balandat, Brian Karrer, Jacob Gardner, Roman Garnett
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our work makes progress on the intractable multi-step formulation of BO through the following methodological and empirical contributions: Improved optimization performance. Using our method, we are able to achieve significant improvements in optimization performance over one-step EI and BINOCULARS on a range of benchmarks, while maintaining competitive wall times. We present our empirical results in Section 7 and conclude in Section 8. We follow the experimental setting of [15], and test our algorithms using the same set of synthetic and real benchmark functions. For brevity, we only present the results on the synthetic benchmarks here; additional results are given in Appendix F. All algorithms are implemented in Bo Torch [1], and we use a GP with a constant mean and a Matérn 5/2 ARD kernel for BO. GP hyperparameters are re-estimated by maximizing the evidence after each iteration. For each experiment, we start with 2d random observations, and perform 20d iterations of BO; 100 experiments are repeated for each function and each method. We measure performance with GAP = (yi y0)/(y y0). All experiments are run on CPU Linux machines; each experiment only uses one core. |
| Researcher Affiliation | Collaboration | Shali Jiang Facebook shalijiang@fb.com Daniel R. Jiang Facebook drjiang@fb.com Maximilian Balandat Facebook balandat@fb.com Brian Karrer Facebook briankarrer@fb.com Jacob R. Gardner University of Pennsylvania jacobrg@cis.upenn.edu Roman Garnett Washington University in St. Louis garnett@wustl.edu |
| Pseudocode | Yes | Algorithm 1: Multi-Step Tree Evaluation |
| Open Source Code | Yes | All details can be found in our accompanying code submission. |
| Open Datasets | No | No concrete access information (specific link, DOI, repository name, formal citation with authors/year, or explicit reference to an established benchmark dataset with specific access details) is provided for the datasets, only a reference to the experimental setting of a prior paper and general mention of synthetic/real functions. While synthetic functions are often well-known, no explicit access details are provided within this paper, and for real functions like 'NN Cancer', no source is given. |
| Dataset Splits | No | The paper describes the setup for Bayesian Optimization, which involves iterative data collection rather than fixed dataset splits common in supervised learning. Therefore, explicit train/test/validation dataset splits with percentages or counts are not applicable or provided in the traditional sense. |
| Hardware Specification | No | The paper states: "All experiments are run on CPU Linux machines; each experiment only uses one core." This provides general information but lacks specific hardware details like CPU model numbers, memory, or GPU specifications. |
| Software Dependencies | No | The paper mentions software like "Bo Torch [1]" and "GPy Torch [8]" but does not specify any version numbers for these software components or any other libraries/solvers. |
| Experiment Setup | Yes | For each experiment, we start with 2d random observations, and perform 20d iterations of BO. GP hyperparameters are re-estimated by maximizing the evidence after each iteration. We use a GP with a constant mean and a Matérn 5/2 ARD kernel for BO. We use GH quadrature to generate samples for approximating the expectation in each stage, with the number of samples fixed heuristically. with number of GH samples m1 = 10, m2 = 5, m3 = 3. We set the budget of 300 function evaluations for optimizing the one-shot objective. |