Practical Adversarial Multivalid Conformal Prediction
Authors: Osbert Bastani, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, Aaron Roth
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We give both theory and an extensive set of empirical evaluations. |
| Researcher Affiliation | Academia | Osbert Bastani1, Varun Gupta1, Christopher Jung2, Georgy Noarov1, Ramya Ramalingam1, Aaron Roth1 1 University of Pennsylvania, 2 Stanford University |
| Pseudocode | Yes | Algorithm 1: MVP(δ, η, m, r) |
| Open Source Code | Yes | Code to replicate our experiments can be found at https://github.com/ProgBelarus/MultiValidPrediction. |
| Open Datasets | Yes | First we replicate an experiment of Tibshirani et al. [2019], in which a synthetic covariate shift (with known propensity scores and known changepoint) is simulated on a UCI dataset. ... datasets derived from 2018 U.S. Census data provided from the Folktables package [Ding et al., 2021]. ... Finally, in Section B.4 we compare MVP to the work of Angelopoulos et al. [2020] on a large-scale Image Net classification task. |
| Dataset Splits | No | The paper discusses data splitting for 'split conformal prediction' (e.g., 'split conformal prediction must split the data into two sets: one for training the regression model and one for calibrating prediction sets'), but it does not provide specific training, validation, or test dataset splits for its own experiments or methodology. |
| Hardware Specification | No | The paper states 'Our experiments are fairly lightweight, and can be run e.g. on a standard 12gb RAM Google Colab account.', which provides memory and computing environment, but does not specify exact GPU or CPU models or processor types. |
| Software Dependencies | No | The paper mentions using a 'predictive model' and 'conformal scores', but it does not specify any software dependencies (e.g., libraries, frameworks, or solvers) with specific version numbers (e.g., 'PyTorch 1.9', 'TensorFlow 2.x'). |
| Experiment Setup | No | The paper states that 'full details can be found in Appendix B' for experiments, and the authors claim to have specified 'training details (e.g., data splits, hyperparameters, how they were chosen)' in their self-assessment. However, the main body of the paper does not contain specific hyperparameter values (e.g., learning rate, batch size, number of epochs) or detailed system-level training settings. |