Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Binary Classification with Karmic, Threshold-Quasi-Concave Metrics
Authors: Bowei Yan, Sanmi Koyejo, Kai Zhong, Pradeep Ravikumar
ICML 2018 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | In this paper, we advance this understanding of binary classification for complex performance measures by identifying two key properties: a so-called Karmic property, and a more technical threshold-quasiconcavity property, which we show is milder than existing structural assumptions imposed on performance measures. Under these properties, we show that the Bayes optimal classifier is a threshold function of the conditional probability of positive class. We then leverage this result to come up with a computationally practical plug-in classifier, via a novel threshold estimator, and further, provide a novel statistical analysis of classification error with respect to complex performance measures. |
| Researcher Affiliation | Academia | 1University of Texas at Austin, Austin, Texas, USA 2University of Illinois at Urbana-Champaign, Champaign, Illinois, USA 3Carnegie Mellon University, Pittsburgh, Pennsylvania, USA. |
| Pseudocode | Yes | Algorithm 1 Two-step Plug-in Classifier for General Metrics |
| Open Source Code | No | The paper does not provide concrete access to source code for the methodology described, nor does it state that code is available. |
| Open Datasets | No | The paper does not provide concrete access information for a publicly available or open dataset. It refers to theoretical models such as the 'Gaussian generative model' and 'β-H older densities' for analytical purposes rather than specific datasets for experimentation. |
| Dataset Splits | No | The paper does not provide specific dataset split information as it focuses on theoretical analysis and does not report on empirical experiments with datasets. |
| Hardware Specification | No | The paper does not provide specific hardware details used for running experiments, as the research presented is theoretical and does not report on empirical experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details (e.g., library or solver names with version numbers), as the research presented is theoretical and does not report on empirical experiments. |
| Experiment Setup | No | The paper does not contain specific experimental setup details (e.g., hyperparameters, training configurations), as the research presented is theoretical and does not report on empirical experiments. |