Mitigating Bias in Adaptive Data Gathering via Differential Privacy
Authors: Seth Neel, Aaron Roth
ICML 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We complement our theoretical results with experiments validating our theory. We run a set of experiments that measure the bias incurred by the standard UCB algorithm in the stochastic bandit setting, contrast it with the low bias obtained by a private UCB algorithm, and show that there are settings of the privacy parameter that simultaneously can make bias statistically insignificant, while having competitive empirical regret with the non-private UCB algorithm. We also demonstrate in the linear contextual bandit setting how failing to correct for adaptivity can lead to false discovery when applying t-tests for non-zero regression coefficients on an adaptively gathered dataset. |
| Researcher Affiliation | Academia | 1Department of Statistics, The Wharton School, University of Pennsylvania 2Department of Computer Science, University of Pennsylvania. |
| Pseudocode | No | The paper describes algorithms such as 'Bandit' and 'Interact Query' in a textual format but does not provide structured pseudocode blocks or a clearly labeled algorithm section. |
| Open Source Code | No | The paper mentions a 'full version' for details and algorithms but does not provide any link or explicit statement about releasing source code for the described methodology. |
| Open Datasets | No | The paper describes the parameters and distributions used to generate synthetic data for its experiments (e.g., 'The K arm means are equally spaced between 0 and 1', 'we observe rewards yi,t N(θ ixit, 1)') but does not use or provide access to a pre-existing, publicly available dataset. |
| Dataset Splits | No | The paper describes running simulations and repeating experiments multiple times (e.g., 'We run UCB and ϵ-private UCB for T rounds', 'We repeat this process 10,000 times') but does not specify traditional training, validation, or test dataset splits. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware (e.g., GPU/CPU models, memory) used to run the experiments. |
| Software Dependencies | No | The paper does not list any specific software or library names with version numbers used for the experiments. |
| Experiment Setup | Yes | In our first stochastic bandit experiment we set K = 20 and T = 500. The K arm means are equally spaced between 0 and 1 with gap = .05, with µ0 = 1. We run UCB and ϵ-private UCB for T rounds with ϵ = .05, and after each run compute the difference between the sample mean at each arm and the true mean. We repeat this process 10,000 times, averaging to obtain high confidence estimates of the bias at each arm. Noting that the observed reduction in bias for ϵ = .05 exceeded that guaranteed by the theory, we run a second experiment with K = 5, T = 100000, = .05, and ϵ = 400, averaging results over 1000 iterations. |