Joint Entropy Search For Maximally-Informed Bayesian Optimization
Authors: Carl Hvarfner, Frank Hutter, Luigi Nardi
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We now evaluate JES on a suite of diverse tasks. We consider three different types of benchmarks: samples drawn from a GP prior, commonly used synthetic test functions [16], and a collection of classification tasks on tabular data using an MLP, provided through HPOBench [10]. For the GP prior tasks, the hyperparameters are known for all methods to evaluate the effect of the acquisition function in isolation. Consequently, we do not use the γ-exploit approach from Sec. 3.5 in this case (i.e., we set γ = 0 in Algorithm 1). For the synthetic and MLP tasks, we marginalize over the GP hyperparameters, and set γ = 0.1. The hyperparameters of the GP prior experiments can be found in Appendix B, and ablation studies on γ in Appendix C. |
| Researcher Affiliation | Collaboration | Carl Hvarfner carl.hvarfner@cs.lth.se Lund University Frank Hutter fh@cs.uni-freiburg.de University of Freiburg Bosch Center for Artificial Intelligence Luigi Nardi luigi.nardi@cs.lth.se Lund University Stanford University DBtune |
| Pseudocode | Yes | Algorithm 1 JES Algorithm |
| Open Source Code | Yes | Our code for reproducing the experiments is available at https://github.com/hvarfner/Joint Entropy Search. |
| Open Datasets | Yes | tuning an MLP model s 4 hyperparameters for 20D iterations on six datasets. These tasks are part of the Open ML1 library of tasks, and the HPO benchmark is provided through the HPOBench [10] suite. |
| Dataset Splits | No | The paper describes the overall experimental setup, including initial design and sequential query selection, but it does not specify explicit train/validation/test dataset splits in terms of percentages or counts for reproducibility. It discusses the use of different GP hyperparameters for different tasks, but not data splitting. |
| Hardware Specification | No | The paper mentions 'The computations were also enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC) at LUNARC partially funded by the Swedish Research Council through grant agreement no. 2018-05973.' However, this is a general reference to a computing infrastructure without providing specific hardware details such as GPU models, CPU types, or memory. |
| Software Dependencies | No | The paper states 'All acquisition functions are all run in the same framework written in MATLAB, created for the original PES implementation by Hernández-Lobato et al. [16].' but does not specify a version number for MATLAB. It also mentions 'GPy: A gaussian process framework in python.' in the references, but does not state its specific usage with a version number in the main text for their experiments. |
| Experiment Setup | Yes | We consider samples from a GP prior for four different dimensionalities: 2D, 4D, 6D, and 12D, with a noise standard deviation of 0.1 for a range of outputs spanning roughly [ 10, 10]. ... For the synthetic and MLP tasks, we marginalize over the GP hyperparameters, and set γ = 0.1. ... All synthetic experiments were run for 50D iterations. In the main paper, we fix the number of MC samples for MES, PES and JES to 100 each. |