Procrastinated Tree Search: Black-Box Optimization with Delayed, Noisy, and Multi-Fidelity Feedback
Authors: Junxiong Wang, Debabrota Basu, Immanuel Trummer10381-10390
AAAI 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We experimentally validate on multiple synthetic functions and hyperparameter tuning problems that PCTS outperforms the stateof-the-art black-box optimization methods for feedbacks with different noise levels, delays, and fidelity. (...) 3. Experimental: We experimentally and comparatively evaluate performance of PCTS on multiple synthetic and real-world hyperparameter optimization problems against the state-of-the-art black-box optimization algorithms (Sec. 4)1. |
| Researcher Affiliation | Academia | Junxiong Wang,1 Debabrota Basu,2 Immanuel Trummer1 1 Dept. of Computer Science, Cornell University, Ithaca, NY, USA 14850 2 Equipe Scool, Inria, UMR 9189 CRISt AL, CNRS, Univ. Lille, Centrale Lille, Lille, France 59000 |
| Pseudocode | Yes | Algorithm 1: PCTS under DNF feedback and with a compatible BANDIT algorithm |
| Open Source Code | Yes | 1Link to our code: https://github.com/jxiw/PCTS |
| Open Datasets | Yes | We illustrate results for three different synthetic functions, Hartmann3 (van der Vlerk 1996), Branin (van der Vlerk 1996), and Currin Exp (Currin et al. 1988) (...) for hyperparameter tuning of SVM on News Group dataset, and XGB and Neural Network on MNIST datasets. |
| Dataset Splits | No | We use corresponding scikit-learn modules (Buitinck et al. 2013) for training all the classifiers. For each tuning task, we plot the median value of crossvalidation accuracy in five runs for 700s, 1700s, and 1800s respectively. While 'cross-validation accuracy' is mentioned, specific details about the splits (e.g., k-fold, train/val/test percentages) are not provided for reproducibility. |
| Hardware Specification | Yes | We run each experiment ten times for 600s on a Mac Book Pro with a 6-core Intel(R) Xeon(R)@2.60GHz CPU (...) We evaluate the aforementioned algorithms on a 32-core Intel(R) Xeon(R)@2.3 GHz server... |
| Software Dependencies | Yes | We implement all baselines in Python (version 2.7). (...) We use corresponding scikit-learn modules (Buitinck et al. 2013) for training all the classifiers. |
| Experiment Setup | Yes | The delay time τ for all synthetic functions is set to four seconds. (...) We set σ2 = 0.02 for algorithms where σ is known, and b = 1 where UCBV and DUCBVare used. (...) The smoothness parameters are computed in a similar manner as POO and MFPOO. For comparison, we keep the delay constant and use wait-and-act versions of delay-insensitive baselines. |