Efficient and Differentiable Conformal Prediction with General Function Classes
Authors: Yu Bai, Song Mei, Huan Wang, Yingbo Zhou, Caiming Xiong
ICLR 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically test our CP-Gen-Recal algorithm (using the practical implementation (8)) on three representative real-data tasks. |
| Researcher Affiliation | Collaboration | Yu Bai Salesforce Research yu.bai@salesforce.com Song Mei UC Berkeley songmei@berkeley.edu Huan Wang & Yingbo Zhou & Caiming Xiong Salesforce Research {huan.wang, yingbo.zhou, cxiong}@salesforce.com |
| Pseudocode | Yes | Algorithm 1 Conformal Prediction with General Function Class (CP-Gen) and Algorithm 2 Conformal Prediction with General Fn. Class and Recalibration (CP-Gen-Recal). |
| Open Source Code | Yes | Code available at https://github.com/allenbai01/cp-gen. |
| Open Datasets | Yes | Our choice of the datasets follows (Feldman et al., 2021). We provide information about these datasets in Table 3. ... Physicochemical properties of protein tertiary structure data set. https://archive.ics.uci.edu/ml/datasets/Physicochemical+Properties+of+Tertiary+Structure. Accessed: January, 2019. |
| Dataset Splits | Yes | All datasets are standardized so that inputs and labels have mean 0 and standard deviation 1, and split into (train, cal, recal, test) with size 70%, 10%, 10%, 10% (varying with the random seed). |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., GPU models, CPU types) used for running the experiments. |
| Software Dependencies | No | The paper mentions software components like '3-layer MLP', 'momentum SGD', and 'Open AI Gym' but does not specify their version numbers or other software dependencies with specific versions. |
| Experiment Setup | Yes | Our network architecture is a 3-layer MLP with width 64 and output dimension 2 (for the lower and upper quantile). We use momentum SGD with initial learning rate 10 3 and momentum 0.9, batch-size 1024, and run the optimization for a max of 10000 epochs. A 10x learning rate decay is performed if the validation loss on Dcal has not decreased in 10 epochs, and we stop the learning whenever the learning rate decay happens for 3 times. ... To solve that optimization problem, we use SGD on (θ, t) with learning rate 0.01 and (ascent on) λ with learning rate 0.1. The batch-size here is 256 and the number of episodes is 1000. |