Efficient Parameter Importance Analysis via Ablation with Surrogates
Authors: Andre Biedenkapp, Marius Lindauer, Katharina Eggensperger, Frank Hutter, Chris Fawcett, Holger Hoos
AAAI 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In our experiments, we demonstrate speed-up factors between 33 and 14 727 for ablation analysis on various configuration scenarios from AI planning, SAT, ASP and mixed integer programming (MIP). |
| Researcher Affiliation | Academia | Andr e Biedenkapp, Marius Lindauer, Katharina Eggensperger, Frank Hutter University of Freiburg {biedenka, lindauer, eggenspk, fh}@cs.uni-freiburg.de Chris Fawcett, Holger H. Hoos University of British Columbia {fawcettc, hoos}@cs.ubc.ca |
| Pseudocode | Yes | A concise summary of ablation analysis is given in Figure 1. Fawcett and Hoos (2016) introduced two variants of ablation analysis. |
| Open Source Code | Yes | Our surrogate-ablation implementation in Python is freely available2 and fully integrated into the algorithm configuration system SMAC. |
| Open Datasets | Yes | As configuration benchmarks, we used the following six benchmarks from the algorithm configuration literature as implemented in AClib (Hutter et al. 2014a)3; details for these benchmarks are shown in Table 1: |
| Dataset Splits | Yes | Table 1: Overview of used configuration benchmarks from AClib. #P is the number of parameters, κ is the running time cutoff, #Inst is the number of instances in the training and test set, Budget is the running time required to run SMAC 16 times, and #Data is the number of collected data points. |
| Hardware Specification | Yes | All experiments were executed on a compute cluster equipped with two Intel Xeon E5-2650v2 8-core CPUs, 20 MB L2 cache and 64 GB of (shared) RAM per node, running Ubuntu 14.04 LTS 64 bit. |
| Software Dependencies | Yes | We fully integrated our surrogate-based ablation procedure into the prominent, general-purpose algorithm configuration system SMAC (Hutter, Hoos, and Leyton-Brown 2011b)1. 1In version 3; see http://www.ml4aad.org/smac/ |
| Experiment Setup | Yes | For the configuration part, we used 16 independent SMAC runs and picked the configuration with the best performance on the training instances as the ablation target θtarget. The performance metric was penalized average running time with a commonly used penalization factor of X = 10 (PAR10). Then, to obtain a ground truth for ablation, we ran a full ablation analysis using all training instances. |