Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning
Authors: Dong Bok Lee, Aoxuan Zhang, Byungjoo Kim, Junhyeon Park, Steven Adriaensen, Juho Lee, Sung Ju Hwang, Hae Beom Lee
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We validate our algorithm on established multifidelity HPO benchmarks and show that it outperforms all the previous freezethaw BO and transfer-BO baselines we consider, while achieving a significantly better trade-off between the cost and performance. Our code is publicly available at https://github.com/db-Lee/CFBO. |
| Researcher Affiliation | Collaboration | 1KAIST 2University of Freiburg 3Deep Auto.ai 4Korea University |
| Pseudocode | Yes | Algorithm 1 Cost-Sensitive Freeze-thaw BO. Blue parts correspond to specifics of our method. |
| Open Source Code | Yes | Our code is publicly available at https://github.com/db-Lee/CFBO. |
| Open Datasets | Yes | We evaluate CFBO on three standard LC benchmarks. LCBench [67]: contains learning curves of MLPs trained on multiple tabular datasets, Task Set [37]: provides diverse optimization tasks across domains; we focus on 30 NLP tasks (text classification and language modeling), and PD1 [56]: includes learning curves of modern neural architectures, such as Transformers, trained on large-scale datasets (CIFAR-10/100 [31], Image Net [45], and bioinformatics corpora). |
| Dataset Splits | Yes | We split each benchmark into disjoint training and test tasks for transfer learning of LC extrapolators pθ. Detailed dataset statistics are summarized in Tab. 1, and additional descriptions are provided in Appendix D. ... Training datasets: APSFailure, Amazon employee access, Australian, Fashion-MNIST, KDDCup09 appetency, Mini Boo NE, adult, airlines, albert, bank-marketing, blood-transfusion-service-center, car, christine, cnae-9, connect-4, covertype, credit-g, dionis, fabert, helena. Test datasets: higgs, jannis, jasmine, jungle chess 2pcs raw endgame complete, kc1, kr-vs-kp, mfeat-factors, nomao, numerai28.6, phoneme, segment, shuttle, sylvine, vehicle, volkert. |
| Hardware Specification | Yes | All measurements are conducted on a single A100 GPU. |
| Software Dependencies | No | The paper mentions using Adam optimizer [28], GELU activation [21], and Transformers [54], but does not provide specific version numbers for software libraries like PyTorch or TensorFlow, or for the Syne Tune [46] package mentioned in the implementation details for baselines. |
| Experiment Setup | Yes | We set the maximum budget4 as B = 300 ... with α {0, 2 6, 2 5, 2 4, 2 3, 2 2}. ... For the baselines, we set the threshold δ = 0.2 in Eq. 5, as it performs well on the training split. For CFBO, we also use γ = log0.5 0.2, which corresponds to the adaptive threshold δb = 0.2 when pb = 0.5, to ensure a fair comparison with the baselines. We use β = exp( 1) for all experiments in this paper... We sample 4 training tasks for each iteration, i.e., the size of meta mini-batch is set to 4. We uniformly sample the size C of context points from 1 to 300, and the size of query points Q is set to 2,048. ... The hidden size of each Transformer block dh, the hidden size of feed-forward networks, and the number of layers of Transformer, dropout rate are set 1,024, 2,048, 12, and 0.2, respectively. We use Ge LU activation [21]. We train the extrapolator for 100,000 iterations on training split of each benchmark with Adam [28] optimizer. The ℓ2 norm of the gradient is clipped to 1.0. The learning rate is linearly increased to 2 10 05 for 25,000 iterations (25% of the total iteration), and it is decreased with a cosine scheduling until the end. |