Batch Multivalid Conformal Prediction
Authors: Christopher Jung, Georgy Noarov, Ramya Ramalingam, Aaron Roth
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate the performance of both of our algorithms in an extensive set of experiments. and We design, analyze, and empirically evaluate two algorithms: one for giving group conditional guarantees for an arbitrary collection of groups G, and the other for giving full multivalid coverage guarantees for an arbitrary collection of groups G. |
| Researcher Affiliation | Academia | Anonymous authors Paper under double-blind review - The paper is under double-blind review, so author affiliations are not disclosed. |
| Pseudocode | Yes | Algorithm 1: Batch GCP(f, G, q, D) and Algorithm 2: Batch MVP(f, α, q, G, ρ, S, m) |
| Open Source Code | Yes | The full Python implementations of Batch GCP and Batch MVP can be found in the supplementary zip. Jupyter notebooks that implement each of our experiments are also included in the supplementary zip. |
| Open Datasets | Yes | 10 real datasets derived from US Census data from the 10 largest US States using the Folktables package of Ding et al. (2021). |
| Dataset Splits | Yes | The training data is further split into training data of size 5000 (with which to train a least squares regression model f) and calibration data of size 15000 (with which to calibrate our various conformal prediction methods). and for each state, we split it into 60% training data Dtrain for the income-predictor, 20% calibration data Dcalib to calibrate the conformal predictors, and 20% test data Dtest. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for running experiments. |
| Software Dependencies | No | The paper mentions 'Python implementations' and 'Jupyter notebooks' in its reproducibility statement, but does not provide specific version numbers for Python or any libraries used (e.g., PyTorch, NumPy, scikit-learn). |
| Experiment Setup | Yes | We run Algorithm 1 (Batch GCP) and Algorithm 2 (Batch MVP with m = 100 buckets) to achieve group-conditional and full multivalid coverage guarantees respectively, with respect to G with target coverage q = 0.9. and We run all four algorithms (naive split-conformal, the method of Foygel Barber et al. (2020), Batch GCP, and Batch MVP with m = 300 buckets) with target coverage q = 0.9. |