Self-Calibrating Conformal Prediction
Authors: Lars van der Laan, Ahmed M. Alaa
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Real-data experiments show that our method improves interval efficiency through model calibration and offers a practical alternative to feature-conditional validity. |
| Researcher Affiliation | Academia | Lars van der Laan University of Washington lvdlaan@uw.edu Ahmed M. Alaa UC Berkeley and UCSF amalaa@berkeley.edu |
| Pseudocode | Yes | Algorithm 1 Venn-Abers Calibration; Algorithm 2 Self-Calibrating Conformal Prediction |
| Open Source Code | Yes | A Python implementation of SC-CP is provided in the package Self Calibrating Conformal, available via pip. Code implementing SC-CP and reproducing our experiments is available in the Git Hub repository Self Calibrating Conformal, which can be accessed at the following link: https: //github.com/Larsvanderlaan/Self Calibrating Conformal. |
| Open Datasets | Yes | We analyze the Medical Expenditure Panel Survey (MEPS) dataset [1], supplied by the Agency for Healthcare Research and Quality [13], which was used in [47] for Mondrian CP with fairness applications. We use the preprocessed dataset acquired using the Python package cqr, also associated with [47]. This dataset contains n = 15, 656 observations and d = 139 features... |
| Dataset Splits | Yes | We randomly partition the dataset into three segments: a training set (50%) for model training, a calibration set (30%) for CP, and a test set (20%) for evaluation. |
| Hardware Specification | Yes | The methods implemented in this paper are not computationally intensive and were run in a Jupyter notebook environment on a Mac Book Pro with 16GB RAM and an M1 chip. |
| Software Dependencies | No | The paper mentions using a "Jupyter notebook environment" and refers to "xgboost [12]" as an implementation of gradient boosted regression trees. However, it does not specify version numbers for Python, Jupyter, xgboost, or any other critical libraries (e.g., PyTorch, TensorFlow, scikit-learn) that would be necessary for precise replication. |
| Experiment Setup | Yes | For training the initial model f( ), we use the xgboost [12] implementation of gradient boosted regression trees [20], where maximum tree depth, boosting rounds, and learning rate are tuned using 5-fold cross-validation. ... SC-CP is implemented as described in Alg. 2, using isotonic regression constrained to have at least 20 observations averaged within each constant segment to mitigate overfitting (via the minimum child weight argument of xgboost). The miscoverage level is taken to be α = 0.1. |