Adaptive Sampling for Minimax Fair Classification
Authors: Shubhanshu Shekhar, Greg Fields, Mohammad Ghavamzadeh, Tara Javidi
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We then validate the benefits of adaptively constructing training sets via experiments on synthetic tasks with logistic regression classifiers, as well as on several real-world tasks using convolutional neural networks (CNNs).5 Empirical Results We evaluate the performance of our proposed active sampling algorithm Aopt on both synthetic and real datasets, and compare it with the following baselines: 1) ϵ-greedy scheme of Abernethy et al. (2020), 2) Greedy scheme, which is equivalent to ϵ = 0, 3) Uniform, where an equal number of samples are drawn for each attribute, and 4) Uncurated, where samples are drawn according to the natural distribution of the dataset. |
| Researcher Affiliation | Collaboration | Electrical and Computer Engineering Department at UCSD ({shshekha,grfields,tjavidi}@ucsd.edu) Google Research (ghavamza@google.com) |
| Pseudocode | Yes | Algorithm 1: Optimistic Adaptive Sampling for Minimax Fair Classification. |
| Open Source Code | Yes | We will make the code for these experiments available in the supplemental materials. |
| Open Datasets | Yes | Adult dataset. For the remaining experiments we find that, for properly tuned values of ϵ and c0, both Aopt and ϵ-greedy attain comparable minimum test error. So we omit the ϵ-greedy results for the purpose of clarity, see Appendix C for a detailed comparison of the two algorithms. We now analyze the performance of the remaining algorithms on a dataset from the UCI ML Repository (Dua and Graff, 2017) that is commonly used in the fairness literature: the Adult dataset. To this end we use three image datasets: UTKFace (Zhang et al., 2017), Fashion MNIST (Xiao et al., 2017), and Cifar10 (Krizhevsky and Hinton, 2009) with CNNs. |
| Dataset Splits | Yes | accuracy of the classifier, ˆft, on the attribute s validation set, Dzt, over time. |
| Hardware Specification | No | The paper states 'Experiments for image datasets were run on a single GPU provided by Google Colab or other shared computing resources,' but does not specify the exact GPU model or other hardware components. |
| Software Dependencies | No | The paper mentions using 'logistic regression classifiers' and 'convolutional neural networks (CNNs)' but does not specify any software names with version numbers, such as Python, PyTorch, or TensorFlow versions, or specific library versions. |
| Experiment Setup | Yes | For Aopt, we use UCB given in 6 with c0 = 0.1 and for ϵ-greedy, we use ϵ = 0.1. These values of c0 and ϵ are selected via a grid search. We use the heuristic variants of the Aopt, with c1 = 0, and Greedy algorithms with batch sizes of 50. The CNN architecture, data transforms, and further algorithm parameters are detailed in Appendix E. |