An Efficient Approach for Assessing Hyperparameter Importance
Authors: Frank Hutter, Holger Hoos, Kevin Leyton-Brown
ICML 2014 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conducted experiments with prominent machine learning frameworks and state-of-the-art solvers for combinatorial problems. We show that our methods provide insight into the relationship between hyperparameter settings and performance, and demonstrate that even in very highdimensional cases most performance variation is attributable to just a few hyperparameters. |
| Researcher Affiliation | Academia | Frank Hutter FH@INFORMATIK.UNI-FREIBURG.DE University of Freiburg, Freiburg, GERMANY Holger Hoos HOOS@CS.UBC.CA University of British Columbia, Vancouver, CANADA Kevin Leyton-Brown KEVINLB@CS.UBC.CA University of British Columbia, Vancouver, CANADA |
| Pseudocode | Yes | Algorithm 1: Compute Partitioning(Θ, T , i, Θ(i)) and Algorithm 2: Quantify Importance(Θ, T , K) |
| Open Source Code | Yes | Our implementation, along with a quick start guide showing how to apply it to your own algorithms, is publicly available at www.automl.org/fanova. |
| Open Datasets | Yes | For example, Bayesian optimization found a better instantiation of nine convolutional network hyperparameters than a domain expert, thereby achieving the lowest error reported on the CIFAR-10 benchmark at the time (Snoek et al., 2012). |
| Dataset Splits | No | Not found. While the paper mentions 'cross-validation folds' in the context of hyperparameter optimization generally and for the Auto-WEKA framework, it does not provide specific dataset split percentages, sample counts, or explicit cross-validation setup details for its own experiments in a way that allows direct reproduction of the data partitioning. |
| Hardware Specification | No | Not found. The paper does not provide specific hardware details such as GPU or CPU models, processor types, or memory specifications used for running its experiments. |
| Software Dependencies | No | Not found. While the paper mentions frameworks and solvers like WEKA, SMAC, and CPLEX, it does not provide specific version numbers for these or other ancillary software components used in their experiments, which is required for reproducible description. |
| Experiment Setup | No | Not found. The paper discusses hyperparameters of the machine learning algorithms and solvers it analyzes, but it does not provide specific experimental setup details (e.g., hyperparameters, training configurations, or system-level settings) for the training or execution of its own proposed functional ANOVA and random forest models. |