Cost-aware Bayesian Optimization via the Pandora's Box Gittins Index
Authors: Qian Xie, Raul Astudillo, Peter Frazier, Ziv Scully, Alexander Terenin
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate the proposed acquisition function, termed the Pandora s Box Gittins Index (PBGI), on a comprehensive set of experiments to understand its strengths and weaknesses. On both sufficientlyeasy low-dimensional problems and too-difficult high-dimensional ones, performance is comparable to baselines. On most medium-hard problems of moderate dimension, the proposed acquisition function either matches or outperforms the strongest baselines. Surprisingly, we find this performance carries over to the classical setting with uniform costs. We also discuss limitations, including behavior on problems where baselines are stronger. |
| Researcher Affiliation | Academia | Qian Xie1 Raul Astudillo2 Peter I. Frazier1 Ziv Scully1 Alexander Terenin1 1Cornell University 2Caltech |
| Pseudocode | No | The paper describes algorithms and processes in text but does not include formal pseudocode blocks or figures labeled 'Algorithm' or 'Pseudocode'. |
| Open Source Code | Yes | Code available at: HTTPS://GITHUB.COM/QIANJANEXIE/PANDORABAYESOPT. |
| Open Datasets | No | The paper refers to datasets like 'Pest Control', 'Lunar Lander', and 'Robot Pushing' but does not provide specific access links, DOIs, or formal citations with author/year for public availability. |
| Dataset Splits | No | The paper describes initializing algorithms with 2(d+1) values using a Sobol sequence and using 16 seeds, but does not specify explicit training/validation/test dataset splits (e.g., percentages or sample counts). |
| Hardware Specification | Yes | All computations were run on CPU, with individual experiments ran in parallel on various nodes of the Cornell G2 cluster, each allocated up to 4GB of memory, except for certain exceptions. These exceptions include KG, MSEI, and BMSEI for higher dimensions, which required substantially more memory, up to 32GB. |
| Software Dependencies | Yes | We implement all methods in Bo Torch [5] using Matérn Gaussian processes with smoothness ν = 5/2. |
| Experiment Setup | Yes | For PBGI, we choose the constant hyperparameter to be λ = 10 4. For PBGI-D, we choose the initial value to be λ0 = 10 1 and the constant decay factor β = 2. For both variants, we compute the Gittins indices using 100 iterations of bisection search without any early stopping or other performance and reliability optimizations. |