Uncertainty Estimation by Fisher Information-based Evidential Deep Learning
Authors: Danruo Deng, Guangyong Chen, Yang Yu, Furui Liu, Pheng-Ann Heng
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | As demonstrated empirically, our proposed method consistently outperforms traditional EDL-related algorithms in multiple uncertainty estimation tasks, especially in the more challenging few-shot classification settings. In this section, we conduct extensive experiments to compare the performance of our proposed method with previous methods on multiple uncertainty estimation-related tasks. |
| Researcher Affiliation | Collaboration | 1Department of Computer Science and Engineering, The Chinese University of Hong Kong 2Institute of Medical Intelligence and XR, The Chinese University of Hong Kong 3Zhejiang Lab. Correspondence to: Guangyong Chen <gychen@zhejianglab.com>. |
| Pseudocode | Yes | Algorithm 1 I-Evidential Deep Learning |
| Open Source Code | Yes | The code is available at: https://github.com/danruod/IEDL |
| Open Datasets | Yes | Datasets We evaluate our algorithm on the following image classification datasets: MNIST (Le Cun, 1998), CIFAR10 (Krizhevsky et al., 2009), and mini-Image Net (Vinyals et al., 2016). For OOD detection experiments, we use KMNIST (Clanuwat et al., 2018) and Fashion MNIST (Xiao et al., 2017) for MNIST, the Street View House Numbers (SVHN) (Netzer et al., 2018) and CIFAR100 (Krizhevsky et al., 2009) for CIFAR10, and the Caltech UCSD Birds (CUB) dataset (Wah et al., 2011) for mini Image Net. |
| Dataset Splits | Yes | For all experiments on both datasets, we split the data into train, validation, and test sets. We use a validation loss-based early termination strategy to train up to 200 epochs with a batch size of 64. For the MNIST and CIFAR10 datasets... We use (80%, 20%) to split the training samples into training and validation sets. ... We use (95%, 5%) to split the training samples into training and validation sets. |
| Hardware Specification | No | No specific hardware details (like GPU/CPU models, memory, or cloud instance types) are provided for running the experiments. |
| Software Dependencies | No | The paper mentions adapting code from (Charpentier et al., 2020) and (Ghaffari et al., 2021) and using specific models like VGG16 and Wide ResNet-28-10, but it does not provide specific software dependencies with version numbers (e.g., Python, PyTorch, TensorFlow versions). |
| Experiment Setup | Yes | We use a validation loss-based early termination strategy to train up to 200 epochs with a batch size of 64. The learning rate is set to 0.001 for MNIST and FMNIST, 0.0005 for CIFAR10. The coefficient λ of -|I| is set by grid-search (0.1, 0.05, 0.01, 0.005, 0.001). The last chosen hyperparameter is 0.005 for MNIST, 0.01 for FMNIST and 0.05 for CIFAR10. For the mini-Image Net and tiered-Image Net few-shot classification experiments... The coefficient λ is also set by grid-search on the meta-validation set. Table 6 reports the last chosen hyperparameter for few-shot settings. |