Trading Off Resource Budgets For Improved Regret Bounds
Authors: Thomas Orton, Damon Falck
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results: We benchmark both FPML and OGhybrid on an online black-box hyperparameter optimization problem based on the 2020 Neur IPS BBO challenge [Turner et al., 2021]. We find that both these new algorithms outperform OG for various compute budgets. |
| Researcher Affiliation | Academia | Damon Falck University of Oxford damon.falck@gmail.com Thomas Orton University of Oxford thomas.orton@cs.ox.ac.uk |
| Pseudocode | Yes | Algorithm 1 FPML(B,") Require: N B 1, " > 0. Initialize the cumulative cost C0(a) 0 for each arm a 2 A. for round t = 1, . . . , T do 1. For each arm a 2 A, draw a noise perturbation pt(a) 1 " Exp. 2. Calculate the perturbed cumulative costs for round t 1, Ct 1(a) Ct 1(a) pt(a). 3. Pull the B arms with the lowest perturbed cumulative costs according to Ct 1. Break ties arbitrarily. 4. Update the cumulative costs for each arm, Ct(a) Ct 1(a) + ct(a). end for |
| Open Source Code | No | The paper does not contain any explicit statement or link indicating that the source code for the described methodology is publicly available. |
| Open Datasets | Yes | the optimization problem at each round t 2 [T] is to choose the hyperparameters of either a multi-layer perceptron (MLP) or a lasso classifier for one of 184 classification tasks from the Pembroke Machine Learning Benchmark [Olson et al., 2017] (so T = 368). |
| Dataset Splits | No | No explicit train/validation/test dataset splits are provided for the tasks from the Pembroke Machine Learning Benchmark used in the experiments. The paper mentions running each algorithm setting 100 times, but this relates to statistical evaluation rather than data partitioning for model training/validation. |
| Hardware Specification | No | The paper mentions 'B available CPU cores' in the context of an application scenario but does not provide specific hardware details like GPU/CPU models, processor types, or memory amounts used for running its experiments. |
| Software Dependencies | No | The paper mentions using 'geometric sampling' and the 'Bayesmark package', but it does not specify version numbers for any software components or libraries. |
| Experiment Setup | No | The paper states that the " parameter for the bandit subroutines FPML-partial and Exp3 was set to their theoretically optimal values given B, N and T without any fine-tuning. However, it does not provide the specific numerical values of these parameters or other detailed hyperparameters for the experimental setup. |