Fairness under Covariate Shift: Improving Fairness-Accuracy Tradeoff with Few Unlabeled Test Samples
Authors: Shreyas Havaldar, Jatin Chauhan, Karthikeyan Shanmugam, Jay Nandy, Aravindan Raghuveer
AAAI 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We experimentally verify that optimizing with our loss formulation outperforms a number of state-of-the-art baselines in the pareto sense with respect to the fairness-accuracy tradeoff on several standard datasets. The evaluation of our method against the baselines is done via the trade-off between fairness violation (using EOdds) and error (which is 100 accuracy). All algorithms are run 50 times before reporting the mean and the standard deviation. All experiments are run on single NVIDIA Tesla V100 GPU. |
| Researcher Affiliation | Collaboration | 1Google Research India 2UCLA 3Fujitsu Research India {shreyasjh, karthikeyanvs, araghuveer}@google.com, {chauhanjatin100, jayjaynandy}@gmail.com, |
| Pseudocode | Yes | Algorithm 1: Gradient Updates for the proposed objective to learn fairly under covariate shift |
| Open Source Code | Yes | Our source code is made available for additional reference.1 https://github.com/google/uafcs |
| Open Datasets | Yes | We demonstrate our method on 4 widely used benchmarks in the fairness literature, i.e. Adult, Communities and Crime, Arrhythmia and Drug Datasets with detailed description in (Havaldar et al. 2024) s appendix section A.2 due to space constraints. |
| Dataset Splits | Yes | To construct the validation set, we further split the training subset to make the final train:validation:test ratio as 5 : 1 : 4, where the test is distribution shifted. |
| Hardware Specification | Yes | All experiments are run on single NVIDIA Tesla V100 GPU. |
| Software Dependencies | No | The paper mentions using a 'single NVIDIA Tesla V100 GPU' for experiments but does not provide specific version numbers for software dependencies such as Python, PyTorch, TensorFlow, or CUDA. |
| Experiment Setup | Yes | Here λ1 and λ2 are hyperparameters governing the objectives. C1 and C2 are the constraints. We use alternating gradient updates to solve the above min-max problem. Our entire learning procedure consists of two stages: (1) pre-training F for some epochs with only DS and (2) further training F with (5). The procedure is summarized in Algorithm 1 and a high level architecture is provided in (Havaldar et al. 2024) s A.4. The implementation details of all the methods with relevant hyperparameters are provided in (Havaldar et al. 2024) s A.4. |