Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift
Authors: Alex Chan, Ahmed Alaa, Zhaozhi Qian, Mihaela Van Der Schaar
ICML 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 5. Experiments Empirical evaluations demonstrate that our method performs competitively compared to Bayesian and frequentist approaches in the task of prostate cancer mortality prediction across globally diverse populations. |
| Researcher Affiliation | Academia | 1University of Cambridge, Cambridge, UK 2University of California, Los Angeles, USA 3Alan Turing Institute, London, UK. |
| Pseudocode | No | The paper includes Figure 1 as a 'High-level depiction of our approach' which is a diagram, but it does not contain structured pseudocode or an algorithm block. |
| Open Source Code | No | The paper does not provide any explicit statement or link indicating that the source code for the described methodology is publicly available. |
| Open Datasets | Yes | Our training data consists of 240,486 patients enrolled in the American SEER program (SEER, 2019), while for our target data we consider a group of 10,086 patients enrolled in the British Prostate Cancer UK program (UK, 2019). and we provide further benchmark results on some publicly available data sets from the UCI machine learning repository (Dua & Graff, 2017) in appendix A. |
| Dataset Splits | Yes | Hyperparameter optimisation remains an open problem under covariate shift we used a validation set consisting of 10% of the labelled data selected, not entirely randomly, but based on propensity score matching in order to obtain a set more reflective of the target data. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory specifications) used for running the experiments. |
| Software Dependencies | No | The paper mentions using Adam optimizer and neural network architectures, but does not provide specific version numbers for software dependencies (e.g., Python, PyTorch, TensorFlow, etc.). |
| Experiment Setup | Yes | For all of the neural networks we consider the same architecture of two fully connected hidden layers of 128 units each and tanh activation function. The initial weights are randomly drawn from N(0, 0.1) and all networks are trained using Adam (Kingma & Ba, 2015). Hyperparameter optimisation remains an open problem under covariate shift we used a validation set consisting of 10% of the labelled data selected, not entirely randomly, but based on propensity score matching in order to obtain a set more reflective of the target data. With this, hyperparemeters were selected for all model through grid search. |