Experimental Designs for Heteroskedastic Variance
Authors: Justin Weltz, Tanner Fiez, Alexander Volfovsky, Eric Laber, Blake Mason, houssam nassif, Lalit Jain
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We propose, analyze and empirically evaluate a novel design for uniformly bounding estimation error of the variance parameters, σ2 x. We demonstrate the benefits of this method with two adaptive experimental design problems under heteroskedastic noise, fixed confidence transductive bestarm identification, and level-set identification; proving the first instance-dependent lower bounds in these settings. Lastly, we construct near-optimal algorithms and empirically demonstrate the large improvements in sample complexity gained from accounting for heteroskedastic variance in these designs. |
| Researcher Affiliation | Collaboration | Department of Statistical Science, Duke University, justin.weltz@duke.edu, work conducted at Amazon Amazon.com, USA, fieztann@amazon.com Department of Statistical Science, Duke University, eric.laber@duke.edu Department of Statistical Science, Duke University, alexander.volfovsky@duke.edu Amazon.com, USA, bjmason@amazon.com Meta, USA, houssamn@meta.com Michael G. Foster School of Business, University of Washington, lalitj@uw.edu, work conducted at Amazon |
| Pseudocode | Yes | Algorithm 1: HEAD (Heteroskedasticity Estimation by Adaptive Designs) ... Algorithm 2: (H-RAGE) Heteroskedastic Randomized Adaptive Gap Elimination ... Algorithm 3: Uniform Estimator of Heteroskedastic Variance ... Algorithm 4: Seperate Arm Estimator of Heteroskedastic Variance ... Algorithm 5: (RAGE) Randomized Adaptive Gap Elimination |
| Open Source Code | No | The paper does not contain any explicit statements or links indicating that source code for their methodology is openly available. |
| Open Datasets | No | The paper describes simulation settings and benchmarks (e.g., 'twist on a standard benchmark [46]', 'Define two sets of arms X1, X2 such that x X1 is drawn uniformly from a unit sphere'), but does not provide concrete access (link, DOI, specific citation with author/year) to a publicly available dataset used for training. |
| Dataset Splits | No | The paper discusses simulation experiments and comparisons, but does not provide specific details on training, validation, or test dataset splits (e.g., percentages, counts, or references to predefined splits). |
| Hardware Specification | No | The paper mentions running experiments on 'a dedicated cluster' but does not provide specific hardware details such as CPU/GPU models, memory, or cloud instance specifications. |
| Software Dependencies | No | The paper states, 'we use the Franke-Wolfe method to compute the designs,' but does not specify any software names with version numbers for reproducibility. |
| Experiment Setup | Yes | All algorithms are run at a confidence level of δ = 0.05, and we use the Franke-Wolfe method to compute the designs. ... For a simulation setting with 2 variations in 3 dimensions, we define Σ = diag(0.3, 0.7, 10 3, 10 3, . . .) R7 7 and θ = (0, 0.005, 0.0075, 0.01, 0.1, 0.1, . . .) R7. |