Combining Statistical Depth and Fermat Distance for Uncertainty Quantification
Authors: Hai Vy Nguyen, Fabrice Gamboa, Reda CHHAIBI, Sixin Zhang, Serge Gratton, Thierry Giaccone
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We measure the out-of-domain uncertainty in the prediction of Neural Networks using a statistical notion called Lens Depth (LD) combined with Fermat Distance, which is able to capture precisely the depth of a point with respect to a distribution in feature space, without any distributional assumption. Our method also has no trainable parameter. The method is applied directly in the feature space at test time and does not intervene in training process. As such, it does not impact the performance of the original model. The proposed method gives excellent qualitative results on toy datasets and can give competitive or better uncertainty estimation on standard deep learning datasets compared to strong baseline methods. |
| Researcher Affiliation | Collaboration | 1 Ampere Software Technology 2 Institut de mathématiques de Toulouse 3 Institut de Recherche en Informatique de Toulouse |
| Pseudocode | No | The paper does not contain a figure, block, or section explicitly labeled 'Pseudocode', 'Algorithm', or 'Algorithm X' providing structured steps for a method or procedure in a code-like format. |
| Open Source Code | Yes | Our code can be found at LD-experiment-code. |
| Open Datasets | Yes | We first evaluate our method on the two-moon dataset. Then, we evaluate on 2 benchmarks Fashion MNIST-MNIST and CIFAR10-Tiny Image Net/CIFAR100/SVHN the ability of our method for the detection of OOD. ... For generating two-moon dataset, we use package scikit-learn [22], with noise 0.07, random state 1, and 1000 samples. ... We train classification models on the dataset Fashion MNIST [28]. ... We train the models on the training set CIFAR10 [8] and then the test set of CIFAR10 is considered as in-distribution (ID) data. We use test sets of 3 datasets SVHN [21], CIFAR100 and Tiny-Image Net [10] as OOD data. |
| Dataset Splits | Yes | From training set, we randomly split 80:20 to have training data and validatation. We choose the best model based on accuracy on validation set. |
| Hardware Specification | No | The paper states 'All experiments related to neural networks are implemented in Pytorch 2.0.1+cuda', which implies the use of NVIDIA GPUs. However, it does not specify the exact GPU models (e.g., 'NVIDIA A100', 'RTX 2080 Ti'), CPU models, or any other specific hardware specifications used for running the experiments. |
| Software Dependencies | Yes | All experiments related to neural networks are implemented in Pytorch 2.0.1+cuda, with its default initialization. ... For generating two-moon dataset, we use package scikit-learn [22]. |
| Experiment Setup | Yes | We train model for 160 epochs using Adam optimizer, learning rate 10 3, other parameters are set by default of Pytorch package. ... For Res Net18 without spectral normalization (SN), we use Res Net-18 model implemented by [3] with a minor modification and training scheme of the same authors. More specifically, after Global Average Pooling layer of CNN, we add a layer of output dimension of 25 instead of 256 proposed by [27] before softmax layer. For training model, we use SGD optimizer with nesterov momentum, learning rate 0.1 (decayed by factor of 5 at epochs 60,120 and 160), momentum 0.9, weight decay 5 10 4.Model is trained for 200 epochs. |