DivBO: Diversity-aware CASH for Ensemble Learning
Authors: Yu Shen, Yupeng Lu, Yang Li, Yaofeng Tu, Wentao Zhang, Bin CUI
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirical results on 15 public datasets show that Div BO achieves the best average ranks (1.82 and 1.73) on both validation and test errors among 10 compared methods, including post-hoc designs in recent Auto ML systems and state-of-the-art baselines for ensemble learning on CASH problems. |
| Researcher Affiliation | Collaboration | Yu Shen1, Yupeng Lu1, Yang Li4, Yaofeng Tu3, Wentao Zhang56, Bin Cui12 1Key Lab of High Confidence Software Technologies, Peking University, China 2Institute of Computational Social Science, Peking University (Qingdao), China 3 ZTE Corporation, China 4 Data Platform, TEG, Tencent Inc., China 5 Mila Québec AI Institute 6HEC, Montréal, Canada |
| Pseudocode | Yes | Algorithm 1: Algorithm framework of Div BO. |
| Open Source Code | Yes | Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] |
| Open Datasets | Yes | In addition, we use 15 public classification datasets that are collected from Open ML [40], whose number of samples ranges from 2k to 20k. More details about the datasets are provided in Appendix A.2. [40] J. Vanschoren, J. N. Van Rijn, B. Bischl, and L. Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49 60, 2014. |
| Dataset Splits | Yes | Each dataset is split into three sets, which are the training (60%), validation (20%), and test (20%) sets. |
| Hardware Specification | Yes | Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.4. (Appendix A.4: All experiments are conducted on a cluster with 4 NVIDIA Tesla V100 GPUs and 80 Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz CPUs.) |
| Software Dependencies | Yes | Our framework is implemented on PyTorch 1.8.1 with Python 3.8.5. We use LightGBM (version 3.3.1) as the diversity surrogate. |
| Experiment Setup | Yes | The search space contains 100 hyperparameters in total, and the details of algorithms and feature engineering hyperparameters are provided in Appendix A.3. ... Each dataset is split into three sets, which are the training (60%), validation (20%), and test (20%) sets. ... we set the number of maximum iterations to 250. ... The hyperparameters β and τ are set to 0.05 and 0.2 in Div BO. |