Frugal Optimization for Cost-related Hyperparameters
Authors: Qingyun Wu, Chi Wang, Silu Huang10347-10354
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide strong empirical results in comparison with state-of-the-art HPO methods on large Auto ML benchmarks. On the empirical side, we perform extensive evaluations using a latest Auto ML benchmark (Gijsbers et al. 2019) which contains large scale classification tasks. We also enrich it with datasets from a regression benchmark (Olson et al. 2017) to test regression tasks. Compared to existing random search algorithm and four variations of Bayesian optimization, CFO shows better anytime performance and better final performance in tuning a popular library XGBoost (Chen and Guestrin 2016) and deep neural networks on most of the tasks with a significant margin. |
| Researcher Affiliation | Industry | Qingyun Wu *, Chi Wang*, Silu Huang Microsoft Research {Qingyun.Wu, Wang.Chi, Silu.Huang}@microsoft.com |
| Pseudocode | Yes | Algorithm 1 FLOW2; Algorithm 2 CFO |
| Open Source Code | Yes | CFO is available in an open-source Auto ML library FLAML https://github.com/microsoft/FLAML/tree/main/flaml/tune with all the extensions discussed. |
| Open Datasets | Yes | We perform an extensive experimental study using a latest open source Auto ML benchmark (Gijsbers et al. 2019), which includes 39 classification tasks. We enriched it with 14 regression tasks from PMLB (Olson et al. 2017). All the datasets are available on Open ML. |
| Dataset Splits | Yes | Each task consists of a dataset in 10 folds, and a metric to optimize: Roc-auc for binary tasks, log-loss for multi-class tasks, and r2 score for regression tasks. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., CPU, GPU models, memory, cloud instances) used for running the experiments. |
| Software Dependencies | No | The paper mentions tuning 'XGBoost' and 'deep neural networks' and using 'Open ML' datasets, but it does not specify software dependencies with version numbers (e.g., Python version, specific library versions like PyTorch, TensorFlow, or scikit-learn). |
| Experiment Setup | No | The paper mentions 'tuning 9 hyperparameters for XGBoost' and also evaluating on 'deep neural networks' but does not provide specific hyperparameter values (e.g., learning rate, batch size, number of epochs) or other detailed training settings for these models. |