The Statistical Scope of Multicalibration
Authors: Georgy Noarov, Aaron Roth
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | We make a connection between multicalibration and property elicitation and show that (under mild technical conditions) it is possible to produce a multicalibrated predictor for a continuous scalar property Γ if and only if Γ is elicitable. On the negative side, we show that for non-elicitable continuous properties there exist simple data distributions on which even the true distributional predictor is not calibrated. On the positive side, for elicitable Γ, we give simple canonical algorithms for the batch and the online adversarial setting, that learn a Γ-multicalibrated predictor. This generalizes past work on multicalibrated means and quantiles, and in fact strengthens existing online quantile multicalibration results. To further counter-weigh our negative result, we show that if a property Γ1 is not elicitable by itself, but is elicitable conditionally on another elicitable property Γ0, then there is a canonical algorithm that jointly multicalibrates Γ1 and Γ0; this generalizes past work on mean-moment multicalibration. Finally, as applications of our theory, we provide novel algorithmic and impossibility results for fair (multicalibrated) risk assessment. |
| Researcher Affiliation | Academia | 1Department of Computer and Information Sciences, University of Pennsylvania, Philadelphia, PA, USA. Correspondence to: Georgy Noarov <gnoarov@seas.upenn.edu>, Aaron Roth <aaroth@cis.upenn.edu>. |
| Pseudocode | Yes | Algorithm 1 Batch Multicalibration(Γ, G, m, f, L); Algorithm 2 Joint Multicalibration((Γ0, Γ1), G, m, (f 0, f 1)); Algorithm 3 Batch Multicalibration V (V, G, m, f, α); Algorithm 4 General Algorithm for the Learner that Achieves Sublinear AMF Regret; Algorithm 5 Online Multicalibration(G, V, m) |
| Open Source Code | No | The paper does not contain any explicit statements about releasing source code or provide links to a code repository. |
| Open Datasets | No | The paper is theoretical and does not report on experiments conducted using specific datasets. While it discusses concepts like 'dataset distribution D' and 'i.i.d. sample ˆD Dn', it does not provide concrete access information (e.g., links, DOIs, formal citations) for any publicly available or open datasets used for training. |
| Dataset Splits | No | The paper is theoretical and does not present empirical experiments that would require train/test/validation splits. Therefore, no specific data split information is provided. |
| Hardware Specification | No | The paper is purely theoretical and does not report on any experiments that would require specific hardware. Therefore, no hardware specifications (e.g., GPU models, CPU types, memory) are mentioned. |
| Software Dependencies | No | The paper focuses on theoretical aspects and algorithm design without reporting on empirical experiments. As such, it does not list any specific software dependencies with version numbers (e.g., programming languages, libraries, or solvers) that would be needed for reproducibility. |
| Experiment Setup | No | The paper is theoretical and focuses on algorithm design and mathematical proofs. It does not describe any empirical experiments, and thus no details regarding experimental setup, such as hyperparameters or system-level training settings, are provided. |