Modular Conformal Calibration
Authors: Charles Marx, Shengjia Zhao, Willie Neiswanger, Stefano Ermon
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct an empirical study of MCC on 17 regression datasets. |
| Researcher Affiliation | Academia | 1Computer Science Department, Stanford University. Correspondence to: Charles Marx <ctmarx@stanford.edu>. |
| Pseudocode | Yes | Algorithm 1 Modular Conformal Calibration Input: base predictor f : X R, calibration score ϕ : R Y R and interpolation algorithm ψ Input: calibration dataset (X1, Y1), , (Xn, Yn) Compute calibration scores Si = ϕ(f(Xi), Yi) for i = 1, . . . , n Run the interpolation algorithm q = ψ (S1, , Sn) Return: the CDF predictor H[x](y) = q(ϕ(f(x), y) |
| Open Source Code | No | No explicit statement or link providing access to the source code for the methodology described in this paper was found. |
| Open Datasets | Yes | Most datasets come from the UCI database (Dua & Graff, 2017). |
| Dataset Splits | Yes | For each dataset we allocate 60% of the data to learn the base predictor, 20% for recalibration and 20% for testing. |
| Hardware Specification | No | No specific hardware details (such as GPU/CPU models, processor types, or memory amounts) used for running experiments were provided. |
| Software Dependencies | No | No specific ancillary software details (e.g., library or solver names with version numbers) needed to replicate the experiment were provided. |
| Experiment Setup | Yes | For POINT predictors the output dimension is 1 and we minimize the L2 error. For QUANTILE predictors we use 2, 4, 7, 10 equally spaced quantiles (denoted in the plots as quantile-2, quantile-4, quantile-7, quantile-10)... We optimize the neural network with the pinball loss. For INTERVAL predictors we use the same setup as (Romano et al., 2019)... For DISTRIBUTION predictors the output of the neural network is 2 dimensions, and we interpret the two dimension as the mean / standard deviation of a Gaussian. We optimize the neural network with the negative log likelihood. For ENSEMBLE predictors we use the setup in (Lakshminarayanan et al., 2017) and learn an ensemble of Gaussian distribution predictors. |