On Fairness and Calibration
Authors: Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, Kilian Q. Weinberger
NeurIPS 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Finally, we evaluate these theoretical findings empirically, comparing calibrated notions of nondiscrimination against the (uncalibrated) Equalized Odds framework on several datasets. These experiments further support our conclusion that calibration and error-rate constraints are in most cases mutually incompatible goals. |
| Researcher Affiliation | Academia | Geoff Pleiss , Manish Raghavan , Felix Wu, Jon Kleinberg, Kilian Q. Weinberger Cornell University, Department of Computer Science {geoff,manish,kleinber}@cs.cornell.edu, {fw245,kwq4}@cornell.edu |
| Pseudocode | Yes | Algorithm 1 Achieving Calibration and an Equal-Cost Constraint via Information Withholding |
| Open Source Code | No | The paper does not include an unambiguous statement about releasing code for the work described, nor does it provide a direct link to a source-code repository. |
| Open Datasets | Yes | The Adult Dataset from UCI Machine Learning Repository [28] contains 14 demographic and occupational features for various people... The Heart Dataset from the UCI Machine Learning Repository contains 14 processed features from 906 adults... we modify the predictions made by the COMPAS tool [12], a risk-assessment tool used in practice by the American legal system. |
| Dataset Splits | No | The paper states 'The original classifiers are trained on a portion of the data, and then the new classifiers are derived using a separate holdout set.' and mentions 'holdout set Pvalid' in Algorithm 1, but it does not provide specific percentages, sample counts, or citations to predefined splits needed to reproduce the data partitioning. |
| Hardware Specification | No | The paper does not provide specific hardware details such as exact GPU/CPU models, processor types, or memory amounts used for running its experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details (e.g., library or solver names with version numbers) needed to replicate the experiment. |
| Experiment Setup | No | The paper mentions specific parameters for the cost function ('rfp = 1 and rfn = 3') in one experiment, but it does not provide general experimental setup details such as concrete hyperparameter values, training configurations, or system-level settings for model training. |