Online conformal prediction with decaying step sizes
Authors: Anastasios Nikolas Angelopoulos, Rina Barber, Stephen Bates
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We include two experiments: an experiment on the Elec2 dataset (Harries et al., 1999) where the data shows significant distribution shift over time, and an experiment on Imagenet (Deng et al., 2009) where the data points are exchangeable. Figures 1 and 2 display the results of the experiment for the Elec2 data and the Imagenet data, respectively. |
| Researcher Affiliation | Academia | 1Department of Electrical Engineering and Computer Science, University of California at Berkeley, Berkeley CA USA 2Department of Statistics, University of Chicago, Chicago IL USA 3Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge MA USA. |
| Pseudocode | No | The paper describes the method and updates using mathematical equations and prose (e.g., equation 4), but does not include a formally labeled 'Pseudocode' or 'Algorithm' block. |
| Open Source Code | Yes | Code to reproduce these experiments is available at https://github.com/aangelopoulos/ online-conformal-decaying. |
| Open Datasets | Yes | The Elec2 (Harries et al., 1999) dataset is a time-series of 45312 hourly measurements of electricity demand in New South Wales, Australia. ... The Imagenet (Deng et al., 2009) is a standard computer vision dataset of natural images. |
| Dataset Splits | Yes | We use even-numbered time points as the time series, and odd-numbered time points as a holdout set for estimating Coveraget(qt). ... We use 45000 points for the time series, and the remaining 5000 points as a holdout set for estimating Coveraget(qt). |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., GPU/CPU models, memory, cloud instance types) used for running the experiments. |
| Software Dependencies | No | The paper does not explicitly list specific software dependencies with version numbers (e.g., Python version, library versions like PyTorch or TensorFlow). |
| Experiment Setup | Yes | The experiments are run with two different choices of step size for online conformal: first, a fixed step size (ηt 0.05); and second, a decaying step size (ηt = t 1/2 ϵ with ϵ = 0.1). All methods are run with α = 0.1. ... In these experiments, like in the main text, we set ϵ = 0.1. ... Change points are identified when at least Nmiscoverage consecutive miscoverage events or Ncoverage events are observed in a row (we set these constants to 10 and 30 by default, respectively). |