Collaborative Bayesian Optimization with Fair Regret
Authors: Rachael Hwee Ling Sim, Yehong Zhang, Bryan Kian Hsiang Low, Patrick Jaillet
ICML 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically demonstrate the benefits (e.g., increased fairness) of our algorithm using synthetic and real-world datasets. This section empirically evaluates the performance and properties of our collaborative BO algorithm using a benchmark function: Hartmann-6d, and three real-world collaborative black-box optimization problems: (a) hyperparameter tuning of a logistic regression (LR) model with a mobile sensor dataset (Anguita et al., 2013), (b) hyperparameter tuning of a convolutional neural network (CNN) with federated extended MNIST (FEMNIST) dataset (Caldas et al., 2018), and (c) mobility demand hotspot discovery on a traffic dataset (Chen et al., 2013). |
| Researcher Affiliation | Collaboration | 1Department of Computer Science, National University of Singapore, Republic of Singapore 2Peng Cheng Laboratory, People s Republic of China 3Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, USA. Correspondence to: Bryan Kian Hsiang Low <lowkh@comp.nus.edu.sg>. |
| Pseudocode | No | The paper refers to 'Algorithm 1 of Appendix A' but the provided text does not contain the algorithm itself. |
| Open Source Code | No | The paper does not contain an explicit statement about the release of source code or a link to a code repository for the methodology described. |
| Open Datasets | Yes | Hartmann-6d, and three real-world collaborative black-box optimization problems: (a) hyperparameter tuning of a logistic regression (LR) model with a mobile sensor dataset (Anguita et al., 2013), (b) hyperparameter tuning of a convolutional neural network (CNN) with federated extended MNIST (FEMNIST) dataset (Caldas et al., 2018), and (c) mobility demand hotspot discovery on a traffic dataset (Chen et al., 2013). |
| Dataset Splits | No | The paper mentions 'validation accuracy' and 'validation set' but does not specify how the datasets were split into training, validation, and test sets with percentages or sample counts. For example, 'The output of the objective function is the validation accuracy of the LR model' and 'The output of the objective function is the validation accuracy of the CNN model'. |
| Hardware Specification | No | The paper does not provide any specific hardware details such as GPU models, CPU types, or memory used for running the experiments. |
| Software Dependencies | No | The paper does not specify any software dependencies with version numbers (e.g., Python, PyTorch, TensorFlow versions, or specific library versions). |
| Experiment Setup | Yes | For each experiment, we repeat 10 runs of BO with different random seeds and plot the standard error as the error bars. The objective function is modeled as a GP with kxx chosen to be a squared exponential kernel. We set αt = c1d(Pn i=1 w2 i ) log (c2t) in (5). We consider two settings: (i) Fix c1: c1 and c2 are fixed across different ρ s, and (ii) Vary c1: c2 is fixed but c1 varies for different ρ s s.t. the ratio of p Pn i=1 w2 i in the exploration term to Pn i=1 wi (i.e., total weight of the exploitation term) is a constant. Details on the choices of c1 and c2 are shown in Appendix C.1. |