DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization
Authors: Noor Awad, Neeratyoy Mallik, Frank Hutter
IJCAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Section 5 then presents comprehensive experiments on artificial toy functions, surrogate benchmarks, Bayesian neural networks, reinforcement learning, and 13 different tabular neural architecture search benchmarks, demonstrating that DEHB is more effective and robust than a wide range of other HPO methods |
| Researcher Affiliation | Collaboration | Noor Awad1 , Neeratyoy Mallik1 , Frank Hutter1,2 1Department of Computer Science, University of Freiburg, Germany 2Bosch Center for Artificial Intelligence, Renningen, Germany |
| Pseudocode | Yes | Algorithm 2 in Appendix B shows the pseudocode for HB with the SH subroutine. ... full pseudocode can be found in Algorithm 3 in Appendix C. |
| Open Source Code | Yes | Our reference implementation of DEHB is available at https: //github.com/automl/DEHB. |
| Open Datasets | Yes | We use a broad collection of publicly-available HPO and NAS benchmarks: all HPO benchmarks that were used to demonstrate the strength of BOHB [Falkner et al., 2018]2 and also a broad collection of 13 recent tabular NAS benchmarks represented as HPO problems [Awad et al., 2020]. ... Two regression datasets from UCI5 were used for the experiments: Boston Housing and Protein Structure. http://archive.ics.uci.edu/ml/index.php ... NAS-Bench-101 [Ying et al., 2019], NAS-Bench-1shot1 [Zela et al., 2020], NAS-Bench-201 [Dong and Yang, 2020] and NAS-HPO-Bench [Klein and Hutter, 2019]. |
| Dataset Splits | No | The paper mentions 'validation regret' and uses various benchmarks (e.g., NAS-Bench-101, NAS-Bench-201, UCI datasets) which typically have predefined splits. However, it does not explicitly state the specific dataset split percentages, sample counts, or the methodology for creating the splits (e.g., random seed, stratified splitting) within the provided text, nor does it refer to specific citations for these splits within the context of training, validation, and test. |
| Hardware Specification | No | The paper discusses computational efficiency and runtime overhead but does not provide specific hardware details such as GPU models, CPU models, memory amounts, or detailed computer specifications used for running its experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details, such as library or solver names with their corresponding version numbers, needed to replicate the experiment. |
| Experiment Setup | Yes | Details for the hyperparameter values of the used algorithms can be found in Appendix D.1. We use the same parameter settings for mutation factor F = 0.5 and crossover rate p = 0.5 for both DE and DEHB. The population size for DEHB is not user-defined but set by its internal Hyperband component while we set it to 20 for DE following [Awad et al., 2020]. |