Bayesian Adaptation for Covariate Shift
Authors: Aurick Zhou, Sergey Levine
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate our method on a variety of distribution shifts for image classification, including image corruptions, natural distribution shifts, and domain adaptation settings, and show that our method improves both accuracy and uncertainty estimation. In our experiments, we aim to analyze how our test-time adaptation procedure in BACS performs when adapting to various types of distribution shift, in comparison to prior methods, in terms of both the accuracy of the adapted model, and its ability to estimate uncertainty and avoid over-confident but incorrect predictions. |
| Researcher Affiliation | Academia | Aurick Zhou, Sergey Levine Department of Electrical Engineering and Computer Sciences University of California, Berkeley {aurick,svlevine}@berkeley.edu |
| Pseudocode | Yes | Algorithm 1 Bayesian Adaptation for Covariate Shift (BACS) |
| Open Source Code | Yes | Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Included in supplemental materials. |
| Open Datasets | Yes | We first evaluate our method on CIFAR-10-C, CIFAR-100-C, and Image Net-C [Hendrycks and Dietterich, 2019], where distribution-shifted datasets are generated by applying different image corruptions at different intensities to the test sets of CIFAR10, CIFAR100 [Krizhevsky, 2012], and Image Net [Deng et al., 2009] respectively. |
| Dataset Splits | No | The paper mentions using training data and test data, and refers to standard datasets, but does not explicitly detail the training/validation dataset splits within the main text. |
| Hardware Specification | No | The paper mentions 'compute support from Google Cloud and the Tensorflow Research Cloud (TFRC) program' but does not specify the exact hardware (e.g., GPU models, CPU types) used for the experiments in the main text. |
| Software Dependencies | No | The paper mentions using 'Tensorflow Research Cloud (TFRC) program' but does not specify software dependencies with version numbers in the main text. |
| Experiment Setup | Yes | For all methods utilizing ensembles, we use ensembles of 10 models, and report results averaged over the same 10 seeds for the non-ensembled methods. While adapting our networks using the entropy loss, we also allow the batch normalization statistics to adapt to the target distribution. For methods that optimize on the test distribution, we report results after one epoch of adaptation unless otherwise stated. |