PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces
Authors: Shuhei Watanabe, Archit Bansal, Frank Hutter
IJCAI 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In a series of experiments, we first verify that our algorithm correctly provides global and local HPI in a toy function. Then we demonstrate that our algorithm takes only less than a second for 105 data points while the prior f-ANOVA [Hutter et al., 2014] would take more than a week. To provide a solid picture of how to use our method, we perform analysis on JAHS-Bench-201 [Bansal et al., 2022] |
| Researcher Affiliation | Academia | Shuhei Watanabe , Archit Bansal and Frank Hutter Department of Computer Science, University of Freiburg, Germany {watanabs,bansala,fh}@cs.uni-freiburg.de |
| Pseudocode | Yes | Algorithm 1 Local PED-ANOVA D = {(xn, f(xn))}N n=1 (Dataset to analyze), γ, γ (User-defined quantiles of top domains) 1: See Appendices E.2, E.3 for practical usages 2: Sort D in ascending order by f 3: |Dγ| 2 and |Dγ | 2 must hold 4: Pick the top-γ and -γ quantile observations Dγ, Dγ 5: for d = 1, . . . , D do 6: Count occurrences of unique values c(n) d 7: Build KDEs pd( |Dγ), pd( |Dγ ) by Eq. (15) 8: Compute vγ d by Eq. (16) 9: return {vγ d}D d=1 |
| Open Source Code | Yes | To facilitate reproducibility, our implementation is available at https://github.com/nabenabe0928/local-anova/. |
| Open Datasets | Yes | To provide a solid picture of how to use our method, we perform analysis on JAHS-Bench-201 [Bansal et al., 2022] |
| Dataset Splits | No | The paper uses the JAHS-Bench-201 as a dataset for analysis, querying it for 'validation accuracy', but does not describe train/validation/test splits for their own method's development or evaluation. |
| Hardware Specification | Yes | Furthermore, all experiments were run on the hardware with Intel Core i7-10700 and we used the f-ANOVA implementation with the default parameter setting by Optuna |
| Software Dependencies | No | The paper mentions using 'Optuna' and 'XGBoost models' but does not provide specific version numbers for these or any other software dependencies. |
| Experiment Setup | Yes | In this experiment, we discretized the HPs with nd = 1001 and all samples were drawn from the uniform distribution. We constructed the dataset D in Algorithm 1 by querying JAHS-Bench-201 for the validation accuracy, i.e. f(x), of N lattice points, where N = 41,343,750 and f(x1, x2, x3, x4) = sum(d=1 to 4) wd(xd) xd^2 ... and W := {Wd}3 d=0 = {50, 5^-1, 5^-2, 5^-3}. |