Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
A/B/n Testing with Control in the Presence of Subpopulations
Authors: Yoan Russac, Christina Katsimerou, Dennis Bohle, Olivier Cappé, Aurélien Garivier, Wouter M. Koolen
NeurIPS 2021 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We illustrate the efficiency of the proposed strategy with numerical simulations on synthetic and real data collected from an A/B/n experiment. |
| Researcher Affiliation | Collaboration | Yoan Russac CNRS, Inria, ENS Université PSL EMAIL Christina Katsimerou Booking.com EMAIL Dennis Bohle Booking.com EMAIL Olivier Cappé CNRS, Inria, ENS Université PSL EMAIL Aurélien Garivier UMPA, CNRS Inria, ENS Lyon EMAIL Wouter M. Koolen Centrum Wiskunde & Informatica EMAIL |
| Pseudocode | No | The paper describes the algorithm steps in text but does not provide a formal pseudocode block or algorithm figure. |
| Open Source Code | Yes | Code at https://gitlab.com/ckatsimerou/abc_s_public |
| Open Datasets | No | The paper mentions using "real data collected from an A/B/n experiment" and synthetically generated data, but does not provide access information (link, DOI, citation) for a publicly available dataset. |
| Dataset Splits | No | The paper does not explicitly provide training/validation/test dataset splits needed to reproduce the experiment. |
| Hardware Specification | No | The paper does not specify any hardware used for running the experiments (e.g., GPU models, CPU types, or cloud instance details). |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers (e.g., library names with versions). |
| Experiment Setup | Yes | In our second experiment 1, we generated 3000 Bernoulli bandit instances with K = 2 and a random number of subpopulations J between 2 and 10. Each subpopulation-arm s mean µa,i is drawn uniformly at random from [0, 1], and the subpopulation frequency vector α is drawn from a Dirichlet(10) distribution. |