On Statistical Bias In Active Learning: How and When to Fix It
Authors: Sebastian Farquhar, Yarin Gal, Tom Rainforth
ICLR 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We first verify that RLURE and RPURE remove the bias introduced by active learning and examine the variance of the estimators. We do this by taking a fixed function whose parameters are independent of Dtrain and estimating the risk using actively sampled points. ... We consider two settings: an inflexible model (linear regression) on toy but non-linear data and an overparameterized model (convolutional Bayesian neural network) on a modified version of MNIST with unbalanced classes and noisy labels. |
| Researcher Affiliation | Academia | University of Oxford, OATML, Department of Computer Science; Department of Statistics |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide an explicit statement or link to open-source code for the methodology. |
| Open Datasets | Yes | We actively classify MNIST and Fashion MNIST images using a convolutional Bayesian neural network (BNN)... |
| Dataset Splits | Yes | 1000 validation points were then removed from this pool to allow early stopping. |
| Hardware Specification | Yes | Computing Infrastructure Nvidia RTX 2080 Ti |
| Software Dependencies | No | Table 1 lists 'Amsgrad (Reddi et al., 2018)' as the optimization algorithm and 'Radial BNN Variational Inference (Farquhar et al., 2020)' as the approximate inference algorithm, but does not provide version numbers for general software dependencies like Python, PyTorch, or CUDA. |
| Experiment Setup | Yes | The details of the hyperparameters used for training are provided in Table 1. ... Learning rate 5 10 4 Batch size 64 Variational training samples 8 Variational test samples 8 Variational acquisition samples 100 Epochs per acquisition up to 100 (early stopping patience=20), with 1000 copies of data Starting points 10 Points per acquistion 1 Acquisition proposal distribution q(im; i1:m 1, Dpool) = e T si P e T si Temperature: T 10,000 Scoring scheme: s BALD (M.I. between θ and output distribution) Variational Posterior Initial Mean He et al. (2016) Variational Posterior Initial Standard Deviation log[1 + e 4] Prior N(0, 0.252) |