Precision-Recall versus Accuracy and the Role of Large Data Sets
Authors: Brendan Juba, Hai S. Le4039-4048
AAAI 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | A corollary of this observation is that a larger number of examples is necessary and sufficient to address class imbalance, a finding we also illustrate empirically. We illustrate this finding empirically: We observe that for several standard off-the-shelf methods and several data sets featuring significant class imbalance, these imbalance-correcting methods do not provide reliable improvements in precision and recall, but adding more data always does significantly improve precision and recall. |
| Researcher Affiliation | Academia | Brendan Juba, Hai S. Le Washington University in St. Louis 1 Brookings Dr., St. Louis, MO 63130 bjuba@wustl.edu, hsle@wustl.edu |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper mentions using existing open-source libraries (scikit-learn, imbalanced-learn) for their experiments but does not provide a link to or explicitly state the release of their own source code for the methodology described in the paper. |
| Open Datasets | Yes | In this experiment, we ran all methods on three data sets from the UCI repository, including all of the data sets that were used by Liu et al. (2006). In addition, we also added a large data set (about 1 million examples) that was used for a virtual screening task in drug discovery (Garnett et al. 2015). |
| Dataset Splits | Yes | In the first experiment, all data sets were stratified into ten folds. We selected data in one fold as small training set, eight folds as large training set, and the last fold as test set. For K-Nearest Neighbor, we performed validation to determine k. |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., CPU, GPU models, memory) used for running the experiments. It only mentions software libraries used. |
| Software Dependencies | No | The paper states, 'we used the Python scikit-learn library (Pedregosa et al. 2011) with the imbalanced-learn package (Lemaître, Nogueira, and Aridas 2016)', but it does not specify the version numbers for these libraries or Python. |
| Experiment Setup | Yes | For the preprocessing algorithms, we used the following settings of our hyperparameters: we set the sampling strategy to float , and left other hyperparameters default for the imbalanced-learn package implementation. For K-Nearest Neighbor, we performed validation to determine k. k was set to 5 for most of the data sets, except for letter (k = 3). For the other classifiers, we set class weight to balanced and left all other hyperparameters as default. |