Robust Yet Efficient Conformal Prediction Sets
Authors: Soroush H. Zargarbashi, Mohammad Sadegh Akhondzadeh, Aleksandar Bojchevski
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 7. Experiments For evasion, we compare CAS with RSCP (Gendler et al., 2021). Even though the original RSCP is not able to handle sparse or discrete data, we extend it and use it as an additional baseline (see C). There are no baselines for poisoning. Since both RSCP and CAS have the same guaranteed coverage we focus on two main metrics: the average size of prediction sets (or efficiency) and the empirical coverage. |
| Researcher Affiliation | Collaboration | Soroush H. Zargarbashi 1 Mohammad Sadegh Akhondzadeh 2 Aleksandar Bojchevski 2 1CISPA Helmholtz Center for Information Security 2University of Cologne. |
| Pseudocode | Yes | Algorithm 1 CDF-Aware Sets (CAS, Evasion) qα = Quant (α; {ˆs(x, y))(x,y) Dcal} Clean quantile Compute scdf(x, y), e.g. with Eq. 10 Upper bound Return Cα = {y : scdf(x, y) qα} Conservative set |
| Open Source Code | Yes | Our code and experiments are in the github repository soroushzargar/CAS. |
| Open Datasets | Yes | We evaluate our method on two image datasets: CIFAR-10 (Krizhevsky, 2009) and Image Net (Deng et al., 2009), and one node-classification (graph) dataset Cora-ML (Mc Callum et al., 2004). |
| Dataset Splits | Yes | The GNN is trained with 20 nodes per class with stratified sampling as the training set and similarly sampled validation set. |
| Hardware Specification | Yes | We run all our experiments both on CPU (Intel(R) Xeon(R) Platinum 8368 CPU @ 2.40GHz) and, and on GPU (NVIDIA A100-SXM4-40GB). |
| Software Dependencies | No | We based our implementation on Py Torch (Paszke et al., 2019) and Pytorch Geometric (Fey & Lenssen, 2019). |
| Experiment Setup | Yes | We used Res Net-110 and Res Net-50 pretrained on CIFAR-10 and Image Net with noisy data augmentation from Cohen et al. (2019). We trained a GCN model (Kipf & Welling, 2017) for node classification. All models are trained on data augmented with noise. The GNN is trained with 20 nodes per class with stratified sampling as the training set and similarly sampled validation set. Noise variance settings used were σ = 0.25 for CIFAR-10 and σ = 0.5 for Image Net. During the Monte Carlo sampling, each datapoint was processed through 104 iterations to calculate the expected probability or mean. For these attacks, we employed the projected gradient descent (PGD) attack (Madry et al., 2017), using an alpha value of 0.1 across 40 iterations. Specifically, we used a perturbation addition probability (p+) of 0.01 and a deletion probability (p-) of 0.6. For the training process, we employed 20 node labels per class for training and similar number of nodes for validation. We conducted the training over 1,000 epochs. |