Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors
Authors: Ali Harakeh, Steven L. Waslander
ICLR 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | All probabilistic object detectors are trained on the COCO (Lin et al., 2014) training data split. For testing, the COCO validation dataset is used as in-distribution data. Following recent recommendations for evaluating the quality of predictive uncertainty estimates (Ovadia et al., 2019), we also test our probabilistic object detectors on shifted data distributions. We construct 3 distorted versions of the COCO validation dataset (C1, C3, and C5) by applying 18 different image corruptions introduced by Hendrycks & Dietterich (2019) at increasing intensity levels [1, 3, 5]. Figures 2 and 3 show the results of evaluating the classification and regression predictive distributions for true positives, localization errors, and false positives under dataset shift and using proper scoring rules. |
| Researcher Affiliation | Academia | Ali Harakeh Institute for Aerospace Studies University of Toronto ali.harakeh@utoronto.ca Steven L. Waslander Institute for Aerospace Studies University of Toronto stevenw@utias.utoronto.ca |
| Pseudocode | No | The paper provides detailed descriptions of the methods and experimental procedures in narrative text and figures, but it does not include any explicit pseudocode blocks or algorithm listings. |
| Open Source Code | Yes | Code for all models, evaluation, and datasets is available at: https://github.com/asharakeh/probdet.git. |
| Open Datasets | Yes | All probabilistic object detectors are trained on the COCO (Lin et al., 2014) training data split. To test on natural dataset shift, we use Open Images data (Kuznetsova et al., 2020) to create a shifted dataset with the same categories as COCO and an out-of-distribution dataset that contain none of the 80 categories found in COCO. |
| Dataset Splits | Yes | All probabilistic object detectors are trained on the COCO (Lin et al., 2014) training data split. For testing, the COCO validation dataset is used as in-distribution data. |
| Hardware Specification | Yes | Each Faster RCNN model takes 3 days to train using 2 P-100 GPUs. On the same setup, Retina Net models take 4 days to finish training. Training for 50 epochs takes 4 days using 4 T-4 GPUs. |
| Software Dependencies | No | The implementation of DETR, Retina Net, and Faster RCNN models is based on original Py Torch implementations available under the Detectron2 (Wu et al., 2019) object detection framework. All models are trained using a fixed random seed, which is shared across all APIs (numpy, torch, detectron2, etc...). While the software frameworks are named, specific version numbers for PyTorch, Detectron2, or NumPy are not provided, which is necessary for reproducibility. |
| Experiment Setup | Yes | We train our probabilistic extensions of those models using 2 GPUS with a learning rate of 0.0025 for Retina Net and 0.005 for Faster RCNN. Both models are trained for 270000 iterations, and the learning rate is dropped by a factor of 10 at 210000 and then again at 250000 iterations. The value of 100 as a base for the exponent was chosen using hyperparameter tuning. |