Bayesian Optimization under Stochastic Delayed Feedback
Authors: Arun Verma, Zhongxiang Dai, Bryan Kian Hsiang Low
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments on synthetic and real-life datasets verify the performance of our algorithms. |
| Researcher Affiliation | Academia | 1Department of Computer Science, National University of Singapore, Republic of Singapore. |
| Pseudocode | Yes | GP-UCB-SDF UCB-based Algorithm for BO-SDF |
| Open Source Code | Yes | Our code is released at https: //github.com/daizhongxiang/BO-SDF. |
| Open Datasets | Yes | In the SVM hyperparameter tuning experiment, the diabetes diagnosis dataset can be found at https://www.kaggle. com/uciml/pima-indians-diabetes-database. [...] In the experiment on hyperparameter tuning for CNN, we use the MNIST dataset and follow the default training-testing sets partitions given by the Py Torch package. [...] In the hyperparameter tuning experiment for LR, the breast cancer dataset we have adopted can be found at https: //archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic). [...] We use the tabular benchmark dataset for SVM hyperparameter tuning from the work of Wistuba et al. (2015), which consists of the validation accuracy evaluated on a discrete domain of six SVM hyperparameters for 50 different classification tasks. [...] The tabular benchmark on hyperparameter tuning of SVM is introduced by the work of (Wistuba et al., 2015) and can be found at https://github.com/wistuba/TST. |
| Dataset Splits | Yes | We use 70% of the dataset as the training set and the remaining 30% as the validation set. For every evaluated hyperparameter configuration, we train the SVM using the training set with the particular hyperparameter configuration and then evaluate the learned SVM on the validation set, whose validation accuracy is reported as the observations. |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU or CPU models, memory, or cloud instance types used for running its experiments. It only vaguely refers to 'machines deployed in the experiment'. |
| Software Dependencies | No | The paper mentions using the 'Py Torch package' but does not specify its version number or any other software dependencies with their versions. |
| Experiment Setup | Yes | In all experiments and for all methods, we use the SE kernel for the GP and optimize the GP hyperparameters by maximizing the marginal likelihood after every 10 iterations. In all experiments where we report the simple regret (e.g., Fig. 1, Fig. 2, etc.), we calculate the simple regret in an iteration using only those function evaluations which have converted (i.e., we ignore all pending observations). [...] We tune the penalty parameter within the range of [10 4, 10] and the RBF kernel parameter within [10 4, 10]. [...] We tune the batch size (within [128, 512]), the learning rate (within [10 6, 1]) and the learning rate decay (within [10 6, 1]). |