MFES-HB: Efficient Hyperband with Multi-Fidelity Quality Measurements
Authors: Yang Li, Yu Shen, Jiawei Jiang, Jinyang Gao, Ce Zhang, Bin Cui8491-8500
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The empirical studies on the real-world Auto ML tasks demonstrate that MFES-HB can achieve 3.3 8.9 speedups over the state-of-the-art approach BOHB. |
| Researcher Affiliation | Collaboration | Yang Li,1,4 Yu Shen,1,4 Jiawei Jiang,2 Jinyang Gao,5 Ce Zhang,2 Bin Cui1,3 1Key Laboratory of High Confidence Software Technologies (MOE), School of EECS, Peking University, Beijing, China 2Department of Computer Science, Systems Group, ETH Zurich, Switzerland 3Institute of Computational Social Science, Peking University (Qingdao), China 4AI Platform, Kuaishou Technology, Beijing, China 5Alibaba Group, Hangzhou, China |
| Pseudocode | Yes | Algorithm 1 Pseudo code for Hyperband. Algorithm 2 Pseudo code of MFES-HB. Algorithm 3 Pseudo code for Sample in MFES-HB. |
| Open Source Code | No | We will open-source the code of MFES-HB and the benchmarks on Github after the review process is done. |
| Open Datasets | Yes | Task Datasets |X| R Bhpo Type Unit FCNet MNIST 10 81 5h #Iterations 0.5 epoch Res Net CIFAR-10 6 81 28h #Iterations 2 epochs XGBoost Covtype 8 27 7.5h Data Subset 1/27 #samples Auto ML 10 Datasets 110 27 4h Data Subset 1/27 #samples |
| Dataset Splits | Yes | In each experiment, we randomly divided 20% of the training dataset as the validation set, tracked the wall clock time (including optimization overhead and evaluation cost), and stored the lowest validation error after each evaluation. |
| Hardware Specification | Yes | All experiments are conducted on CentOS Linux release 7.6.1810 with 64GB of memory and 12 CPUs (Intel(R) Xeon(R) Gold 6133 CPU @ 2.50GHz), and 4 NVIDIA Tesla P100 GPUs (16GB). |
| Software Dependencies | Yes | We use SMAC3 (v0.12.0) and BOHB (v0.7.0) for our experiments. |
| Experiment Setup | Yes | As recommended by HB and BOHB, η is set to 3 for the HB-based methods, and ρ = 0.2. In MFES-HB, we implemented the MFES based on the probabilistic random forest from the SMAC package; the parameter θ used in weight discrimination operator is set to 3. |