A Bandit Approach to Sequential Experimental Design with False Discovery Control
Authors: Kevin G. Jamieson, Lalit Jain
NeurIPS 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | through simulations we show a substantial performance improvement over uniform sampling and an adaptive elimination style algorithm. ... 4 Experiments The distribution of each arm equals i = N(µi, 1) where µi = µ0 = 0 if i 2 H0, and µi > 0 if i 2 H1. We consider three algorithms: i) uniform allocation with anytime BH selection as done in Algorithm 1, ii) successive elimination (SE) (see Appendix G)5 that performs uniform allocation on only those arms that have not yet been selected by BH, and iii) Algorithm 1 (UCB). ... The first panel shows an empirical estimate of E[ |St\H1| |H1| ] at each time t for each algorithm, averaged over 1000 trials. ... The right four panels show the number of samples each algorithm takes before the true positive rate exceeds 1 δ = .95, relative to the number of samples taken by UCB, for various parameterizations. |
| Researcher Affiliation | Collaboration | Kevin Jamieson , , Lalit Jain {jamieson,lalitj}@cs.washington.edu Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, and Optimizely, San Francisco, CA |
| Pseudocode | Yes | Algorithm 1 An algorithm for identifying arms with means above a threshold µ0 using as few samples as possible subject to false alarm and true discovery conditions. The set St is designed to control FDR at level δ. The set Rt is designed to control FWER at level δ. |
| Open Source Code | No | The paper mentions that an algorithm faithful to the theory was implemented in production at Optimizely, and provides a link to Optimizely's product page [12], which is not a direct link to the open-source code for the specific methodology described in the paper. |
| Open Datasets | No | The paper describes a simulated data distribution rather than using or providing access to a publicly available or open dataset. |
| Dataset Splits | No | The paper describes a simulation setup involving repeated trials and does not specify traditional train/validation/test dataset splits. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, memory) used for running the experiments are mentioned in the paper. |
| Software Dependencies | No | The paper describes a formula used but does not provide specific software dependencies (libraries, solvers) with version numbers. |
| Experiment Setup | Yes | Here we present the sample complexity for TPR+FDR with δ = 0.05 and different parameterizations of µ, n, |H1]. ... Algorithm 1 and the BH selection rule for all algorithms use φ(t, δ) = 2 log(1/δ)+6 log log(1/δ)+3 log(log(et/2)) t from [25, Theorem 8]. In addition, we ran BH at level δ instead of δ/(6.4 log(36/δ)) as discussed in section 3. |