Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration
Authors: Yuanjie Shi, Subhankar Ghosh, Taha Belkhouja, Jana Doppa, Yan Yan
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Comprehensive experiments on multiple real-world datasets demonstrate that RC3P achieves class-wise coverage and 26.25% reduction in prediction set sizes on average. |
| Researcher Affiliation | Academia | Yuanjie Shi Washington State University Subhankar Ghosh Washington State University Taha Belkhouja Washington State University Janardhan Rao Doppa Washington State University Yan Yan Washington State University |
| Pseudocode | Yes | Algorithm 1 RC3P Method for Class-Conditional CP |
| Open Source Code | Yes | The code is available at https://github.com/Yuanjie Sh/RC3P. |
| Open Datasets | Yes | We consider four datasets: CIFAR-10, CIFAR-100 [34], mini-Image Net [57], and Food-101 [8] by using the standard training and validation split. |
| Dataset Splits | Yes | Randomly split data into train Dtr and calibration Dcal and train the classifier f on Dtr.we randomly split the original (balanced) validation set into calibration samples and testing samples. |
| Hardware Specification | No | The paper mentions 'Res Net-20 [27]' as the main architecture and 'LDAM proposed by [10]' as the training algorithm, but does not specify any hardware details such as GPU/CPU models or memory used for experiments. |
| Software Dependencies | No | The paper mentions 'PyTorch [44]' but does not provide a specific version number for it or any other software components. |
| Experiment Setup | Yes | To handle imbalanced data, we employ the training algorithm LDAM proposed by [10] that assigns different margins to classes... where all models are trained with 200 epochs... learning rate 0.1, momentum 0.9, and weight decay 2e-4 for 200 epochs and 50 epochs. The batch size is 128. For Food-101, the batch size is 256... We set α = 0.1 as our main experiment setting and also report other experiment results of different α values (See Appendix C.7). |