Asymptotically Optimal and Computationally Efficient Average Treatment Effect Estimation in A/B testing
Authors: Vikas Deep, Achal Bassamboo, Sandeep Kumar Juneja
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Numerical comparisons demonstrate that both policies perform similarly across practical values of ϵ and δ, offering efficient solutions for A/B testing. |
| Researcher Affiliation | Academia | 1Kellogg School of Management, Northwestern University, Evanston, IL 60201 2Ashoka University, Sonipat, Haryana, India. |
| Pseudocode | No | The paper describes the 'Assignment rule', 'Estimation rule', and 'Stopping rule' for Policy P1 and P2 in paragraph form, but it does not present these as a structured pseudocode block or a clearly labeled algorithm. |
| Open Source Code | No | The paper does not provide any statement about releasing source code for the described methodology, nor does it include a link to a code repository. |
| Open Datasets | No | Specifically, we model the outcomes for treatments A and B as exponentially distributed with means µA = 10 and µB = 0.1, respectively. We select ϵ = 0.5 and explore different values of δ, including 10%, 5%, and 1%. For each δ setting, we generate 2000 sample paths and calculate the average outcomes. The paper describes modeling and generating synthetic data for its experiments, rather than using a publicly available dataset with concrete access information. |
| Dataset Splits | No | The paper describes modeling and generating synthetic data for its experiments, and mentions using '2000 sample paths,' but it does not specify any training/validation/test dataset splits. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used to run the experiments, such as CPU or GPU models, memory, or cloud resources. |
| Software Dependencies | No | The paper mentions using specific methods and rules from other research (e.g., 'D-tracking rule introduced in Garivier & Kaufmann (2016)', 'Kaufmann & Koolen (2021) for the choice of β(n, δ)'), but it does not list any general software dependencies with specific version numbers (e.g., Python, PyTorch). |
| Experiment Setup | Yes | We select ϵ = 0.5 and explore different values of δ, including 10%, 5%, and 1%. For each δ setting, we generate 2000 sample paths and calculate the average outcomes. |