Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning
Authors: Joseph Giovanelli, Alexander Tornede, Tanja Tornede, Marius Lindauer
AAAI 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In an experimental study targeting the environmental impact of ML, we demonstrate that our approach leads to substantially better Pareto fronts compared to optimizing based on a wrong indicator pre-selected by the user, and performs comparable in the case of an advanced user knowing which indicator to pick. |
| Researcher Affiliation | Academia | Joseph Giovanelli1, Alexander Tornede2, Tanja Tornede2, Marius Lindauer2 1Alma Mater Studiorum University of Bologna 2Institute of Artificial Intelligence, L3S Research Center, Leibniz University Hannover |
| Pseudocode | No | The paper does not contain a pseudocode block or a clearly labeled algorithm. |
| Open Source Code | Yes | All code including detailed documentation and the technical appendix can be found on Github1. 1https://github.com/automl/interactive-mo-ml |
| Open Datasets | Yes | The concrete hyperparameters our DNN algorithms exposes are defined by LCBench (Zimmer, Lindauer, and Hutter 2021), a well-known multi-fidelity deep learning benchmark, and given in Table 1 in the appendix. LCBench comprises evaluations of over 2000 funnel-shaped MLP neural networks with varying hyperparameters on 35 datasets of the Open ML CC-18 suite (Bischl et al. 2019). |
| Dataset Splits | Yes | The loss function can be used to assess the quality of a hyperparameter configuration by splitting the original dataset D into two disjoint datasets Dtrain and Dtest, where the model is trained only based on Dtrain but evaluated with L on Dtest. Overall, we seek to find the optimal hyperparameter configuration λ P Λ defined as λ P arg min λPΛ L p A p Dtrain, λq , Dtestq . (2) ... To this end, they split up a so-called validation dataset Dval from the training dataset Dtrain to estimate the loss Lp Ap Dtrain, λq, Dvalq and avoid a biased overfit to Dtest. ... We measure the accuracy of a model as the validation accuracy on 33% of the corresponding Open ML CC18 dataset. |
| Hardware Specification | Yes | The evaluations present in LCBench were performend on an Intel Xeon Gold 6242 with a maximum consumption of 150 Wh. |
| Software Dependencies | No | The paper mentions several software tools like SMAC, Optuna, Hyperopt, Hp Band Ster, Syne Tune, but does not specify their version numbers or other software dependencies with versions. |
| Experiment Setup | Yes | The concrete hyperparameters our DNN algorithms exposes are defined by LCBench (Zimmer, Lindauer, and Hutter 2021), a well-known multi-fidelity deep learning benchmark, and given in Table 1 in the appendix. ... We run both IB and PB for a budget of 30 evaluations on each of the datasets for 3 seeds and report the mean and standard deviation over the seeds and datasets. |