Modulating Surrogates for Bayesian Optimization
Authors: Erik Bodin, Markus Kaiser, Ieva Kazlauskaite, Zhenwen Dai, Neill Campbell, Carl Henrik Ek
ICML 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We perform numerous experiments on a range of BO benchmarks and find that our approach improves reliability and performance when faced with challenging objective functions. In this section we will demonstrate the benefit of our approach empirically. As the approach is motivated by robustness to the presence of challenging structures in the objective function, we will test its ability to improve search efficiency on a range of functions exhibiting such structure. Visual examples of functions with typical properties are shown in Figure 3. As we will show, our approach increases reliability in the search when faced with detrimental structure (see Figure 4) that has a large negative impact on traditional surrogates. We compare with and without function modulation (Section 2) implemented as in Section 3 on a popular GP model setup for BO. In addition, we compare the LGP against other methods of handling challenging structure in the objective function, namely (i) a noiseless GP, (ii) a GP with homoscedastic noise, (iii) a GP with heteroscedastic noise and (iv) a non-stationary, Warped GP (Snoek et al., 2014). We follow the standard practice to compare across benchmarks and provide the mean gap estimated over 20 runs as in (Malkomes & Garnett, 2018). Table 1 presents results across a wide range of benchmark functions consisting of the Sig Opt benchmark suite (Mc Court, 2016). Three additional real-world benchmarks (Head et al., 2018; Malkomes & Garnett, 2018; Kaelbling & Lozano-P erez, 2017) are included in the bottom section of the table. |
| Researcher Affiliation | Collaboration | 1University of Bristol, United Kingdom 2Siemens AG, Germany 3Technical University of Munich, Germany 4University of Bath, United Kingdom 5Spotify Research, United Kingdom 6University of Cambridge, United Kingdom. |
| Pseudocode | No | The paper describes methods and models but does not provide any pseudocode or algorithm blocks. |
| Open Source Code | No | The paper cites external tools and frameworks (e.g., 'GPy Opt', 'Spearmint package') but does not provide a direct link to the authors' own implementation code for the methodology described in the paper. |
| Open Datasets | Yes | We perform the comparisons on benchmarks from (Mc Court, 2016; Head et al., 2018) using the default domains provided by respective benchmark, detailed in the supplement. Optimization test functions. https://github.com/sigopt/evalset, 2016. scikitoptimize/scikit-optimize: v0.5.2, March 2018. URL https://doi.org/10.5281/zenodo.1207017. |
| Dataset Splits | No | The paper describes evaluation on benchmarks using an 'evaluation budget' and mentions 'initial random points', but it does not specify explicit training/validation/test dataset splits (e.g., percentages or counts) for reproducibility. |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., GPU/CPU models, memory, cloud instances) used for running the experiments. |
| Software Dependencies | No | The paper mentions software components like 'Matérn 5/2 kernel', 'GPy Opt (GPy Opt, 2016)', and 'Spearmint package (Snoek et al., 2014)', but it does not provide specific version numbers for these dependencies, which are necessary for reproducible descriptions. |
| Experiment Setup | Yes | We use the Mat ern 5/2 kernel for all surrogates, the expected improvement acquisition function (where not otherwise stated) and Bayesian hyperparameter marginalisation as in (Snoek et al., 2012). Specifically, σh U({0.1d, 0.01d, 0}) where d = Q, the length of the diagonal of the unit Qdimensional hypercube. We found that this approach performed well empirically and is applied consistently across all our experiments where not otherwise specified. |