PAC Prediction Sets Under Covariate Shift
Authors: Sangdon Park, Edgar Dobriban, Insup Lee, Osbert Bastani
ICLR 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the effectiveness of our approach on covariate shifts based on Domain Net and Image Net. Our algorithm satisfies the PAC constraint, and gives prediction sets with the smallest average normalized size among approaches that always satisfy the PAC constraint. |
| Researcher Affiliation | Academia | Sangdon Park Dept. of Computer & Info. Science PRECISE Center University of Pennsylvania sangdonp@seas.upenn.edu Edgar Dobriban Dept. of Statistics & Data Science The Wharton School University of Pennsylvania dobriban@wharton.upenn.edu Insup Lee Dept. of Computer & Info. Science PRECISE Center University of Pennsylvania lee@cis.upenn.edu Osbert Bastani Dept. of Computer & Info. Science PRECISE Center University of Pennsylvania obastani@seas.upenn.edu |
| Pseudocode | Yes | Algorithm 1 PS-W: an algorithm using the robust RSCP bound in (20) ... Algorithm 2 PS: an algorithm using the CP bound in (3) ... Algorithm 3 PS-C: an algorithm using the CP bound in (3) with ε/b ... Algorithm 4 PS-R: an algorithm using the RSCP bound in (6) ... Algorithm 5 PS-M: an algorithm using the RSCP bound in (6) along with IWs rescaling |
| Open Source Code | Yes | Related code is released6. 6https://github.com/sangdon/pac-ps-w |
| Open Datasets | Yes | We demonstrate the effectiveness of our approach on covariate shifts based on Domain Net and Image Net. |
| Dataset Splits | Yes | For each source-target distribution pair, we split each the labeled source data and unlabeled target data into train and calibration sets. ... We split the dataset into 409,832 training, 88,371 calibration, and 88,372 test images. |
| Hardware Specification | No | The paper does not explicitly state the specific hardware (e.g., GPU models, CPU types) used for running its experiments. |
| Software Dependencies | No | The paper mentions using a 'deep neural network score function f based on Res Net101' and 'stochastic gradient descent (SGD)' but does not provide specific version numbers for software libraries or dependencies (e.g., PyTorch, TensorFlow versions). |
| Experiment Setup | Yes | For neural network training, we run stochastic gradient descent (SGD) for 100 epochs with an initial learning rate of 0.1, decaying it by half once every 20 epochs. ... Parameters are m = 20, 000, ε = 0.1, and δ = 10 5. ... For the grid search in line 12 of Algorithm 1, we increase τ by 10 7 until the bound URSCP exceeds 1.5ε. |