Multi-Fidelity Multi-Armed Bandits Revisited
Authors: Xuchuang Wang, Qingyun Wu, Wei Chen, John C.S. Lui
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We report the numerical comparisons between both procedures in Figure 1. The detailed setup of the simulations is given in Appendix B. Remark 3.5 (Tightness of cost complexity bounds). The first term of cost complexity upper bound for EXPLORE-A in Eq.(7) matches the cost complexity lower bound in Eq.(2) up to a constant when m k = m k and H = H (i.e., when µ(M) 1 and µ(M) 2 are close to their ground truth values). The cost complexity upper bound of EXPLORE-B in Eq.(8) matches the lower bound with an additional P m M λ(m)/λ(1) coefficient when m k = m k and H = H. Remark 3.6 (Comparison to classic MAB s sample complexity). If we reduce our cost complexity upper bound result in MF-MAB to classic (single-fidelity) MAB, i.e., letting M = 1, λ(m) = 1, then both cost complexity upper bounds reduce to O(P k(1/ 2 k) log(1/δ)) where k := µ1 µk, which is exactly the classic sample complexity upper bound for (single-fidelity) BAI [27, 21]. Appendices B and C also detail simulation setups and results, indicating empirical analysis. |
| Researcher Affiliation | Collaboration | Xuchuang Wang Chinese University of Hong Kong xcwang@cse.cuhk.edu.hk Qingyun Wu Pennsylvania State University qingyun.wu@psu.edu Wei Chen Microsoft Research weic@microsoft.com John C.S. Lui Chinese University of Hong Kong cslui@cse.cuhk.edu.hk |
| Pseudocode | Yes | Algorithm 1 LUCB Framework for Multi-Fidelity BAI... Algorithm 2 EXPLORE Procedures... Algorithm 3 Elimination for MF-MAB... Algorithm 4 EXPLORE-C Procedures |
| Open Source Code | No | The paper does not include an explicit statement about releasing source code for the described methodology, nor does it provide any links to a code repository. |
| Open Datasets | Yes | According to benchmarked results in two recent benchmarks for multi-fidelity hyperparameter optimization, including HPOBench [5] and YAHPO Gym [29], under the typically used fidelity dimension, including the number of epochs, the training data sample, etc., the maximum distance from the terminal validation loss often decreases with the increase of resources, i.e., ζ(m) decreases with the increase of m. |
| Dataset Splits | No | The paper conducts simulations of a Multi-Fidelity Multi-Armed Bandit model, which are parameterized by µ, ζ, and λ values. It does not describe experiments on an empirical dataset that would require training, validation, or test splits. While it mentions 'validation loss' in the context of hyperparameter optimization applications, it does not provide specific data splitting information for its own reported experimental setups. |
| Hardware Specification | No | The paper focuses on theoretical analysis and simulations, and does not specify any hardware details (e.g., GPU/CPU models, memory) used for running these simulations. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers required to replicate the experiments. |
| Experiment Setup | Yes | In Figure 1, we report the cost complexities of Algorithm 1 with EXPLORE-A and Algorithm 1 with EXPLORE-B (let µ(M) 1 = 0.95 and µ(M) 2 = 0.75). We set the confidence parameter δ as 0.05, 0.1, 0.15, 0.2, 0.25 respectively in comparing the performance of both procedures. For each simulation, we run 100 trials, plot their cost complexities mean as markers and their deviation as shaded regions. We present the parameters of MF-MAB instances of Figures 1a and 1b in Tables 1 and 2 respectively. |