Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
Authors: Youngseog Chung, Willie Neiswanger, Ian Char, Jeff Schneider
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide a thorough experimental evaluation of our methods, which includes a high dimensional uncertainty quantification task in nuclear fusion. |
| Researcher Affiliation | Academia | Youngseog Chung Machine Learning Department Carnegie Mellon University Pittsburgh, PA 15213 youngsec@cs.cmu.edu Willie Neiswanger Department of Computer Science Stanford University Stanford, CA 94305 neiswanger@cs.stanford.edu Ian Char Machine Learning Department Carnegie Mellon University Pittsburgh, PA 15213 ichar@cs.cmu.edu Jeff Schneider Robotics Institute Carnegie Mellon University Pittsburgh, PA 15213 schneide@cs.cmu.edu |
| Pseudocode | Yes | Algorithm 1 MAQR 1: Input: Train data {xi, yi}N i=1, trained regression model ˆf(x) 2: Calculate residuals ϵi = yi ˆf(xi), i [N], and denote the residual dataset R = {xi, ϵi}N i=1 3: Initialize D 4: for i = 1 to N do 5: Di CONDQUANTILESESTIMATORS(R, i) (Algorithm 2) 6: D D Di 7: end for 8: Use D to fit a regression model ˆg ˆg : (x, p) 7 ϵ 9: Output: ˆf + ˆg |
| Open Source Code | Yes | Code is available at https://github.com/Youngseog Chung/calibrated-quantile-uq. |
| Open Datasets | Yes | We demonstrate the performances of our proposed methods on the standard 8 UCI datasets [2], and on a real-world problem in nuclear fusion. |
| Dataset Splits | Yes | For each dataset, we use 80% of the data for training, 10% for validation, and 10% for testing. |
| Hardware Specification | No | The paper does not specify the hardware used for running experiments, such as GPU models, CPU types, or cloud computing instances. |
| Software Dependencies | No | The paper mentions using Python-based tools and neural networks but does not provide specific version numbers for software dependencies like Python, PyTorch/TensorFlow, or CUDA. |
| Experiment Setup | Yes | Our neural network models consist of 3 hidden layers with 128 neurons each, and use ReLU activation functions for the hidden layers. We use Adam optimizer with learning rate 1e-3, batch size 64, and weight decay 1e-5. |