Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Supervising the Multi-Fidelity Race of Hyperparameter Configurations
Authors: Martin Wistuba, Arlind Kadra, Josif Grabocka
NeurIPS 2022 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the significant superiority of Dy HPO against state-of-the-art hyperparameter optimization methods through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse architectures (MLP, CNN/NAS, RNN). |
| Researcher Affiliation | Collaboration | Martin Wistuba Amazon Web Services, Berlin, Germany EMAIL Arlind Kadra University of Freiburg, Freiburg, Germany EMAIL Josif Grabocka University of Freiburg, Freiburg, Germany EMAIL |
| Pseudocode | Yes | Algorithm 1 DYHPO Algorithm |
| Open Source Code | Yes | Our implementation of DYHPO is publicly available.3 (Footnote 3: https://github.com/releaunifreiburg/Dy HPO) |
| Open Datasets | Yes | LCBench: A learning curve benchmark [Zimmer et al., 2021]... Task Set: A benchmark that features diverse tasks Metz et al. [2020]... NAS-Bench-201: A benchmark consisting of 15625 hyperparameter configurations representing different architectures on the CIFAR-10, CIFAR-100 and Image Net datasets Dong and Yang [2020]. |
| Dataset Splits | No | The paper describes the benchmarks used (LCBench, Task Set, NAS-Bench-201) and their characteristics, but does not explicitly state the training/validation/test dataset splits used for the experiments conducted in this paper, nor does it refer to specific predefined splits within those benchmarks that they utilized. |
| Hardware Specification | Yes | We ran all of our experiments on an Amazon EC2 M5 Instance (m5.xlarge). |
| Software Dependencies | No | The paper does not explicitly provide a list of specific software dependencies with their version numbers (e.g., Python, PyTorch, TensorFlow, or other libraries with version numbers). |
| Experiment Setup | Yes | For DYHPO, we use a constant learning rate of 0.1 for training the kernel parameters, and we train for 100 iterations per step. For all methods, we use a single learning rate of 0.001. |