Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Augmenting Automated Spectrum Based Fault Localization for Multiple Faults
Authors: Prantik Chatterjee, Jose Campos, Rui Abreu, Subhajit Roy
IJCAI 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 5 Empirical Analysis We implement the proposed multiverse analysis approach in a tool called ARTEMIS. To evaluate the effectiveness of our proposed approach, we experimentally evaluate the following research questions: RQ1: What are the optimal choices of hyperparameters for ARTEMIS? RQ2: How does ARTEMIS compare against the most popular SBFL metrics? |
| Researcher Affiliation | Academia | Prantik Chatterjee1 , Jos e Campos2,3 , Rui Abreu2,4 and Subhajit Roy1 1Indian Institute of Technology Kanpur, India 2Faculty of Engineering of University of Porto, Porto, Portugal 3LASIGE, Faculdade de Ciencias, Universidade de Lisboa, Lisboa, Portugal 4INESC-ID, Porto, Portugal |
| Pseudocode | Yes | Algorithm 1: EXPLORER(A, E, n, S, µ, β, p) ... Algorithm 2: SIMULATE(A, E, C, S, p) ... Algorithm 3: MERGE(L , n) |
| Open Source Code | Yes | We provide an appendix with additional experimental results and the source code of ARTEMIS in https://github.com/prantikchatterjee/Artemis IJCAI23.git |
| Open Datasets | Yes | We have performed our experiments on six java project repositories, Chart, Closure, Lang, Math, Mockito and Time from the DEFECTS4J benchmark suite [Just et al., 2014]. In each fold, we set aside the spectrums from one project as test data and search for the best hyperparameters over the spectrums from the remaining five projects. |
| Dataset Splits | No | The paper describes using cross-validation for hyperparameter selection on the training data, but does not explicitly define a separate 'validation' dataset split with that specific terminology. |
| Hardware Specification | No | The paper does not provide specific hardware details such as CPU/GPU models, memory, or detailed computer specifications used for running its experiments. |
| Software Dependencies | No | The paper mentions using EVOSUITE, ULYSIS, and DDU but does not provide specific version numbers for these or other software dependencies. |
| Experiment Setup | Yes | Our multiverse analysis (Algorithm 1) uses the following hyperparameters: µ, β, the underlying SBFL metric S and p. To select the best hyperparameter combination and to show that the configuration generalizes, we use k-fold cross validation. We employ a grid search to select the best value of hyperparameters. We search for µ in a range from 1 to 20. For β, we restrict the search in {5, 10, 15, 20, . . . , 100}. |