REDUCR: Robust Data Downsampling using Class Priority Reweighting
Authors: William Bankes, George Hughes, Ilija Bogunovic, Zi Wang
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we present empirical results to showcase the performance of REDUCR on large-scale vision and text classification tasks. REDUCR significantly improves worst-class test accuracy (and average accuracy), surpassing state-of-the-art methods by around 15%. |
| Researcher Affiliation | Collaboration | William Bankes Department of Computer Science University College London william.bankes.21@ucl.ac.uk George Hughes Department of Computer Science University College London Ilija Bogunovic Department of Electrical Engineering University College London i.bogunovic@ucl.ac.uk Zi Wang Google Deep Mind wangzi@google.com |
| Pseudocode | Yes | Algorithm 1 REDUCR for robust online batch selection |
| Open Source Code | Yes | Code available at: https://github.com/williambankes/REDUCR. |
| Open Datasets | Yes | We use CIFAR10 [Krizhevsky et al., 2012], CINIC10 [Darlow et al., 2018], Clothing1M [Xiao et al., 2015], the Multi-Genre Natural Language Interface (MNLI), and the Quora Question Pairs (QQP) datasets from the GLUE NLP benchmark [Wang et al., 2019]. The Image datasets were sourced from pytorch via the torchvision datasets package https://pytorch.org/vision/stable/datasets.html, the NLP datasets were sourced from huggingface, https://huggingface.co/datasets/nyu-mll/glue. |
| Dataset Splits | Yes | Each dataset is split into a labelled training, validation and test dataset (for details see Appendix A.5), the validation dataset is used to train the class-irreducible loss models and evaluate the class-holdout loss during training. |
| Hardware Specification | Yes | All models were trained on GCP NVIDIA Tesla T4 GPUs. |
| Software Dependencies | No | The networks are optimised with Adam W [Loshchilov and Hutter, 2019] and the default Pytorch hyperparameters are used for all methods except CINIC10 for which the weight decay is set to a value of 0.1. For the NLP dataset we use the bert-base-uncased [Devlin et al., 2019] model from Hugging Face [Wolf et al., 2020] |
| Experiment Setup | Yes | Unless stated otherwise 10% of batch Bt is selected as the small batch bt, and we set η = 1e 4. γ = 9 is used when training each of the amortised class-irreducible loss models on the vision datasets and γ = 4 for the NLP datasets. For full details of the experimental setup see Appendix A.5. |