Federated Conformal Predictors for Distributed Uncertainty Quantification
Authors: Charles Lu, Yaodong Yu, Sai Praneeth Karimireddy, Michael Jordan, Ramesh Raskar
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our results demonstrate a practical approach to incorporating meaningful uncertainty quantification in distributed and heterogeneous environments. We provide code used in our experiments https://github. com/clu5/federated-conformal. and Thorough empirical evaluations and ablations under data heterogeneity on several benchmark computer vision and medical imaging datasets. |
| Researcher Affiliation | Academia | 1MIT Media Lab, Massachusetts Institute of Technology, Cambridge, USA 2Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, USA. |
| Pseudocode | Yes | To this end, we present our algorithm in Algorithm 1. Algorithm 1 Federated Conformal Prediction (FCP) |
| Open Source Code | Yes | We provide code used in our experiments https://github. com/clu5/federated-conformal. |
| Open Datasets | Yes | Our experiments use several computer vision datasets (Fashion MNIST, CIFAR-10, and CIFAR-100) and medical imaging datasets (Derma MNIST, Path MNIST, Tissue MNIST, and Fitzpatrick17K). (Groh et al., 2021), (Xiao et al., 2017), (Krizhevsky et al., 2009), (Yang et al., 2023). |
| Dataset Splits | Yes | to estimate a threshold ˆτ on a held-out calibration dataset Dcal = {(Xi, Yi)}n i=1 X Y that is assumed to be exchangeable with unseen test data (Xtest, Ytest). and For each experiment, we report metrics over ten trials, where the calibration and test sets are randomly split evenly in each trial. |
| Hardware Specification | No | The paper mentions model architectures (e.g., Efficient-Net-B1, Res Net-14) but does not provide specific details about the hardware used for experiments, such as GPU/CPU models or cloud instance types. |
| Software Dependencies | No | The paper mentions software components like 'Torchvision' and training with 'SGD' but does not specify their version numbers (e.g., PyTorch 1.9, Python 3.8, CUDA 11.1). |
| Experiment Setup | Yes | For Fed Avg, 200 communication rounds with five local epochs. For stage 2 of TCT, we take the Fed Avg model trained with 100 epochs (instead of 200) and additionally train 100 communication rounds with a learning rate of 0.0001 and 500 local steps. We train all models with SGD with 0.9 momentum, a learning rate of 0.01 (0.001 for Fitzpatrick17k), and a minibatch size of 64. Data augmentation, such as random flipping and cropping, was applied to all datasets during training; for the Fitzpatrick17k dataset, random color jittering and rotations were also applied. For RAPS, we set the regularization parameters a = 1 and b = 0.001 (b = 0.00001 for the Fitzpatrick17k dataset). |