Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Fast Cross-Validation via Sequential Testing
Authors: Tammo Krueger, Danny Panknin, Mikio Braun
JMLR 2015 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The paper includes a dedicated section '6. Experiments' where it describes evaluations on 'synthetic and real-world data sets', presents 'mean square error (MSE)' and 'relative speed gain' metrics, and compares its method to 'normal cross-validation'. |
| Researcher Affiliation | Academia | All listed authors are affiliated with 'Technische Universität Berlin Machine Learning Group'. The email domains also point to this academic institution (e.g., 'EMAIL', 'EMAIL'). |
| Pseudocode | Yes | The paper explicitly includes multiple algorithm blocks: 'Algorithm 1 CVST Main Loop', 'Algorithm 2 Find the top configurations via iterative testing', 'Algorithm 3 Check for flop configurations via sequential testing', 'Algorithm 4 Compare performance of remaining configurations', and 'Algorithm 5 Select the winning configuration out of the remaining ones'. |
| Open Source Code | Yes | Additionally, we have released a software package on CRAN named CVST which is publicly available via all official CRAN repositories and also via Git Hub (https://github.com/tammok/CVST). |
| Open Datasets | Yes | For classification we picked a representative choice of data sets from the IDA benchmark repository (see Ratsch et al. 2001). Furthermore we added the first two classes with the most entries of the covertype data set (see Blackard and Dean, 1999). [...] For regression we pick the data used in Donoho and Johnstone (1994) and add the bank32nm, pumadyn32nm and kin32nm of the Delve repository.3 |
| Dataset Splits | Yes | Then we split the data set in half and use one part for training and the other for the estimation of the test error. This process is repeated 50 times to get sufficient statistics for the performance of the methods. |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU or CPU models, processor types, or memory amounts used for running its experiments. It focuses on the methodology and experimental outcomes. |
| Software Dependencies | No | The paper refers to its own 'CVST' software package and mentions the 'WEKA machine learning toolbox' but does not specify version numbers for any external software dependencies or libraries. |
| Experiment Setup | Yes | In all experiments we use a 10 step CVST with parameter settings as described in Section 4.4 (i. e. α = 0.05, αl = 0.01, βl = 0.1, wstop = 3). [...] The explored parameter grid contains 610 equally spaced parameter configurations for each method (log10(σ) {−3, −2.9, . . . , 3} and ν {0.05, 0.1, . . . , 0.5} for SVM/SVR and log10(λ) {−7, −6, . . . −2} for KLR/KRR, respectively). |