Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces
Authors: Shuhei Watanabe, Archit Bansal, Frank Hutter
IJCAI 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In a series of experiments, we ļ¬rst verify that our algorithm correctly provides global and local HPI in a toy function. Then we demonstrate that our algorithm takes only less than a second for 105 data points while the prior f-ANOVA [Hutter et al., 2014] would take more than a week. To provide a solid picture of how to use our method, we perform analysis on JAHS-Bench-201 [Bansal et al., 2022] |
| Researcher Affiliation | Academia | Shuhei Watanabe , Archit Bansal and Frank Hutter Department of Computer Science, University of Freiburg, Germany EMAIL |
| Pseudocode | Yes | Algorithm 1 Local PED-ANOVA D = {(xn, f(xn))}N n=1 (Dataset to analyze), Ī³, Ī³ (User-deļ¬ned quantiles of top domains) 1: See Appendices E.2, E.3 for practical usages 2: Sort D in ascending order by f 3: |DĪ³| 2 and |DĪ³ | 2 must hold 4: Pick the top-Ī³ and -Ī³ quantile observations DĪ³, DĪ³ 5: for d = 1, . . . , D do 6: Count occurrences of unique values c(n) d 7: Build KDEs pd( |DĪ³), pd( |DĪ³ ) by Eq. (15) 8: Compute vĪ³ d by Eq. (16) 9: return {vĪ³ d}D d=1 |
| Open Source Code | Yes | To facilitate reproducibility, our implementation is available at https://github.com/nabenabe0928/local-anova/. |
| Open Datasets | Yes | To provide a solid picture of how to use our method, we perform analysis on JAHS-Bench-201 [Bansal et al., 2022] |
| Dataset Splits | No | The paper uses the JAHS-Bench-201 as a dataset for analysis, querying it for 'validation accuracy', but does not describe train/validation/test splits for their own method's development or evaluation. |
| Hardware Specification | Yes | Furthermore, all experiments were run on the hardware with Intel Core i7-10700 and we used the f-ANOVA implementation with the default parameter setting by Optuna |
| Software Dependencies | No | The paper mentions using 'Optuna' and 'XGBoost models' but does not provide specific version numbers for these or any other software dependencies. |
| Experiment Setup | Yes | In this experiment, we discretized the HPs with nd = 1001 and all samples were drawn from the uniform distribution. We constructed the dataset D in Algorithm 1 by querying JAHS-Bench-201 for the validation accuracy, i.e. f(x), of N lattice points, where N = 41,343,750 and f(x1, x2, x3, x4) = sum(d=1 to 4) wd(xd) xd^2 ... and W := {Wd}3 d=0 = {50, 5^-1, 5^-2, 5^-3}. |