Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks
Authors: Shibo Li, Robert Kirby, Shandian Zhe
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the advantage of our approach on four real-world hyperparameter optimization applications. For evaluation, we examined BMBO-DARN in both synthetic benchmarks and real-world applications. The synthetic benchmark tasks show that given a small number of training examples, our deep auto-regressive model can learn a more accurate surrogate of the target function than other state-of-the-art multi-fidelity BO models. We then evaluated BMBO-DARN on four popular machine learning models (CNN, online LDA, XGBoost and Physics informed NNs) for hyperparameter optimization. |
| Researcher Affiliation | Academia | Shibo Li, Robert M. Kirby, and Shandian Zhe School of Computing, University of Utah Salt Lake City, UT 84112 shibo@cs.utah.edu, kirby@cs.utah.edu, zhe@cs.utah.edu |
| Pseudocode | Yes | Algorithm 1 BMBO-DARN (D, B, M, T, {λm}M m=1 ) ... Algorithm 2 Batch Acquisition({λm}, B, L, G, ξ) |
| Open Source Code | No | The paper does not provide concrete access to source code for the methodology described in this paper, nor does it explicitly state that its own source code is being released. |
| Open Datasets | Yes | We used CIFAR-10 dataset (https://www.cs.toronto.edu/~kriz/cifar.html), from which we used 10K images for training and another 10K for evaluation. We used the implement from the scikit-learn library (https: //scikit-learn.org/stable/). We used 5K documents for training, and 2K for evaluation. We trained an XGBoost model (Chen and Guestrin, 2016) to predict a quantitative measure of the diabetes progression (https://archive.ics.uci.edu/ml/datasets/ diabetes). |
| Dataset Splits | No | The paper does not explicitly provide training/test/validation dataset splits or specify a distinct validation set, only mentioning training and evaluation sets. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., exact GPU/CPU models, processor types, or memory amounts) used for running its experiments. |
| Software Dependencies | No | The paper mentions software like PyTorch, Hamiltorch, and scikit-learn, but does not provide specific version numbers for these dependencies. |
| Experiment Setup | Yes | For each fidelity, we used two hidden layers with 40 neurons and tanh activation. We ran HMC for 5K steps to reach burn in (by looking at the trace plots) and then produced 200 posterior samples with every 10 steps. To generate each sample proposal, we ran 10 leapfrog steps, and the step size was chosen as 0.012. For our method, we set the maximum number of iterations in optimizing the batch acquisition function (see Algorithm 8) to 100 and tolerance level to 10 3. |