Multi-Objective Bayesian Optimization with Active Preference Learning
Authors: Ryota Ozaki, Kazuki Ishikawa, Youhei Kanzaki, Shion Takeno, Ichiro Takeuchi, Masayuki Karasuyama
AAAI 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We perform two types of experiments. First, in 5.1, we evaluate the performance of our MI-based active learning (described in 3.3) that efficiently learns the preference model (2). Next, in 5.2, we evaluate the performance of the entire framework of our proposed method, for which we used a benchmark function and two settings of hyperparameter optimization of machine learning models. |
| Researcher Affiliation | Academia | Ryota Ozaki1, Kazuki Ishikawa1, Youhei Kanzaki1, Shion Takeno3, Ichiro Takeuchi2, 3, Masayuki Karasuyama1 1Nagoya Institute of Technology 2Nagoya University 3RIKEN AIP |
| Pseudocode | Yes | Algorithm 1: Proposed Method 1: procedure MBO-APL 2: Run ACTIVE-PREF-LEARNING in background 3: for t = 1, . . . do 4: Fit GPs to DGP = {(xi, yi)}t i=1 5: xt+1 argmaxx αEI(x) using current p(w | Dpref) 6: Observe (xt+1, yt+1) and DGP DGP (xt+1, yt+1) 7: end for 8: end procedure 9: procedure ACTIVE-PREF-LEARNING 10: for t = 1, . . . do 11: Update p(w | Dpref) with the current Dpref 12: f, f argmaxf,f MI(z PC; w) 13: Query z PC to the DM and add the result to Dpref 14: end for 15: end procedure |
| Open Source Code | No | The paper does not contain any explicit statement about open-source code availability nor does it provide a link to a code repository. |
| Open Datasets | Yes | The datasets for the classifiers are Waveform-5000 (L = 3) and CIFAR-10 (L = 10) (Krizhevsky, Hinton et al. 2009). (...) For two datasets, Adult (Becker and Kohavi 1996) and Bank (Moro, Rita, and Cortez 2012), we prepare 201 candidates of x by taking grid points in [0, 1]. |
| Dataset Splits | No | The paper mentions using a "validation set" for defining objective functions (e.g., "recall of each class on the validation set"), but it does not provide specific split percentages, sample counts, or a clear methodology for how these validation sets were created to allow for reproduction of the data partitioning. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used to run the experiments (e.g., GPU/CPU models, memory specifications). |
| Software Dependencies | No | The paper mentions software like Light GBM, Resnet18, and logistic regression, but it does not provide specific version numbers for any of these or any other ancillary software dependencies, which are necessary for reproducibility. |
| Experiment Setup | Yes | In all experiments, the true utility function (the underlying true DM preference) is defined by (2) with the parameter wtrue, determined through the sampling from the Dirichlet distribution (α = (2, . . . , 2) ). GPs for f(x) employs the RBF kernel. Preference observations are generated with the noise variance σPC = σIR = 0.1. |