Efficient Automatic CASH via Rising Bandits
Authors: Yang Li, Jiawei Jiang, Jinyang Gao, Yingxia Shao, Ce Zhang, Bin Cui4763-4771
AAAI 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The extensive experiments on 30 Open ML datasets demonstrate the superiority of the proposed approach over the competitive baselines. |
| Researcher Affiliation | Collaboration | Yang Li,1 Jiawei Jiang,2 Jinyang Gao,3 Yingxia Shao,4 Ce Zhang,2 Bin Cui1 1Key Laboratory of High Confidence Software Technologies (MOE), School of EECS, Peking University, Beijing, China 2Department of Computer Science, Systems Group, ETH Zurich, Switzerland 3Beijing University of Posts and Telecommunications, Beijing, China 4Alibaba Group, Hangzhou, China |
| Pseudocode | Yes | Algorithm 1 Online algorithm for Rising Bandit Input: ML algorithm candidates A = {A1, ..., AK}, the total time steps T, and one unit of HPO resource ˆb. 1: Initialize Scand = {1, 2, ..., K}, t = 0. 2: while t < T do 3: for each k Scand do 4: t = t + 1. 5: Pull arm k once: Hk Iterate HPO(Ak,ˆb). 6: Calculate ωk(t) according to Hk. 7: Update ut k(T) = min(yk(t) + ωk(t)(T t), 1). 8: Update lt k(T) = yk(t). 9: end for 10: for i = j Scand do 11: if lt i(T) ut j(T) then 12: Scand = Scand\{j} 13: end if 14: end for 15: end while 16: return the corresponding ML algorithm A and its best hyperparameter configuration. |
| Open Source Code | No | The paper does not provide an explicit statement or link for the open-source code of the methodology described. |
| Open Datasets | Yes | The extensive experiments on 30 Open ML datasets demonstrate the superiority of the proposed approach over the competitive baselines. We considered 30 classification datasets from the Open ML repositories. These datasets are widely used in the related works (Feurer et al. 2015; Efimova, Filchenkov, and Shalamov 2017; Olson and Moore 2019; Liu et al. 2019), and the details are listed in Appendix C. |
| Dataset Splits | Yes | For each run, the original dataset will be partition into three subsets: training set, validation set and test set, in the proportion of 64%, 16%, 20%. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., CPU/GPU models, memory, or cloud instance types) used for running its experiments. |
| Software Dependencies | No | The paper mentions software components like SMAC, TPE, Auto-Sklearn, Hyperopt-Sklearn, and TPOT, but does not specify their version numbers or the versions of any underlying programming languages or libraries. |
| Experiment Setup | Yes | It comprises 16 ML classification algorithms with 78 hyperparameters. More details about the space can be found in Appendix B of the supplemental materials. [...] Accuracy is used as the metric of the objective. [...] The parameter C for computing the smooth growth rate is set to 7. |