Nonparametric Uncertainty Quantification for Single Deterministic Neural Network
Authors: Nikita Kotelevskii, Aleksandr Artemenkov, Kirill Fedyanin, Fedor Noskov, Alexander Fishkov, Artem Shelmanov, Artem Vazhentsev, Aleksandr Petiushko, Maxim Panov
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct a series of experiments on image and text classification datasets. |
| Researcher Affiliation | Collaboration | 1Skolkovo Institute of Science and Technology, Moscow, Russia 2Technology Innovation Institute, Abu Dhabi, UAE 3AIRI, Moscow, Russia 4HSE University, Moscow, Russia 5Lomonosov Moscow State University, Moscow, Russia 6Nuro, Inc. 7Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE |
| Pseudocode | Yes | Algorithm 1 NUQ inference algorithm. Input: Training set {(xi, yi)}N i=1, inference point x, bandwidth h Output: Prediction ˆg(x) and uncertainty estimate ˆUt(x) |
| Open Source Code | Yes | The code to reproduce the experiments is available online at https://github.com/stat-ml/NUQ. |
| Open Datasets | Yes | We demonstrate the strong performance of the method in uncertainty estimation tasks on text classification problems and a variety of real-world image datasets, such as MNIST, SVHN, CIFAR-100 and several versions of Image Net. and references for all listed datasets. |
| Dataset Splits | No | The paper states 'The bandwidth h is tuned via cross-validation optimizing the classification accuracy on the training data (see SM, Section C.1).' and 'We discuss this [training details (e.g., data splits, hyperparameters, how they were chosen)] in Supplementary Material, Sections B and G'. However, the main text does not explicitly provide the specific percentages or counts for train/validation/test splits, nor does it explicitly detail how validation was performed beyond bandwidth tuning. |
| Hardware Specification | No | The paper states 'Importantly, it took us approximately 5 minutes to receive uncertainties over all Image Net datasets with a CPU', but it does not specify the CPU model or any other hardware details such as GPU types or memory. The checklist also states 'It is not included in the main paper, but will be addressed in Supplementary material.' |
| Software Dependencies | No | The paper does not provide specific version numbers for software dependencies such as libraries or frameworks used (e.g., 'PyTorch 1.9'). It only mentions general tools or frameworks without versioning, or refers to supplementary material for details. |
| Experiment Setup | No | The paper mentions using 'spectral normalization' and applying NUQ to 'features from the penultimate layer of the model', and that 'the density estimate is given by GMM'. It also mentions 'We use a pre-trained ELECTRA model with 110 million parameters'. However, it explicitly defers 'hyperparameter optimization' and other details to the Supplementary Material (SM, Section G), and does not provide specific hyperparameter values like learning rate or batch size in the main text. |