$\pi$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization
Authors: Carl Hvarfner, Danny Stoll, Artur Souza, Marius Lindauer, Frank Hutter, Luigi Nardi
ICLR 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Further, our experiments show that πBO outperforms competing approaches across a wide suite of benchmarks and prior characteristics. We also demonstrate that πBO improves on the state-of-the-art performance for a popular deep learning task, with a 12.5 time-to-accuracy speedup over prominent BO approaches. |
| Researcher Affiliation | Collaboration | 1Lund University, 2University of Freiburg, 3Federal University of Minas Gerais, 4Leibniz University Hannover, 5Bosch Center for Artificial Intelligence, 6Stanford University |
| Pseudocode | Yes | Algorithm 1 πBO Algorithm |
| Open Source Code | Yes | πBO is publicly available as part of the SMAC (https://github.com/automl/SMAC3) and Hyper Mapper (https://github.com/luinardi/hypermapper) HPO frameworks. Our Spearmint implementation of both πBO and BOWS is available at https://github.com/piboauthors/Pi BO-Spearmint, and our Hyper Mapper implementation is available at https://github.com/piboauthors/ Pi BO-Hypermapper. |
| Open Datasets | Yes | For the Open ML MLP tuning benchmark, we utilize the defaults and search spaces provided in HPOBench (Eggensperger et al., 2021). The 5D parameter space consists of four continous parameters and one integer parameter. We considered two deep learning case studies: segmentation of neuronal processes in electron microscopy images with a U-Net(6D) (Ronneberger et al., 2015), with code provided from the NVIDIA deep learning examples repository (Przemek et al.), and image classification on Image Nette-128 (6D) (Howard, 2019), a light-weight adaptation of Image Net (Deng et al., 2009). |
| Dataset Splits | No | The paper states 'As test splits for both tasks were not available to us, we report validation scores,' and mentions using 'defaults and search spaces provided in HPOBench,' but does not explicitly provide specific training/validation/test split percentages or sample counts for all datasets used in the experiments. |
| Hardware Specification | Yes | For U-Net Medical we used one Ge Force RTX 2080 Ti GPU, whereas for Image Nette-128 we used two Ge Force RTX 2080 Ti GPU s. Also, we used 4 cores and 8 cores respectively, of an AMD EPYC 7502 32-Core Processor. |
| Software Dependencies | No | The paper mentions frameworks like SMAC and Hyper Mapper, and libraries such as Fast AI, but it does not specify version numbers for these or other ancillary software components (e.g., Python version, PyTorch version, specific compiler versions). |
| Experiment Setup | Yes | For all experiments, we run πBO with β = N/10, where N is the total number of iterations... For the Open ML MLP tuning benchmark, we utilize the defaults and search spaces provided in HPOBench (Eggensperger et al., 2021), and construct Gaussian priors for each hyperparameter with their mean on the default value, and a standard deviation of 25% of the hyperparameter s domain. For the DL case studies... for numerical hyperparameters, once again set the standard deviation to 25% of the hyperparameter s domain. For categorical hyperparameters, we place a higher probability on the default. We consider a budget of 50 iterations... In both case studies, we enabled mixed precision training, and for Image Nette-128 to work in conjunction with Spearmint, we had to enable the MKL_SERVICE_FORCE_INTEL environment flag. Tables 1, 2, and 3 detail the search spaces for hyperparameters. |