Optimal PAC Multiple Arm Identification with Applications to Crowdsourcing
Authors: Yuan Zhou, Xi Chen, Jian Li
ICML 2014 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Finally, we report experimental results on both synthetic and real data sets, which demonstrates the superior performance of the proposed algorithm. |
| Researcher Affiliation | Academia | Yuan Zhou, Carnegie Mellon U YUANZHOU@CS.CMU.EDU Xi Chen, UC Berkeley XICHEN@CS.CMU.EDU Jian Li, Tsinghua U LIJIAN83@MAIL.TSINGHUA.EDU.CN |
| Pseudocode | Yes | Algorithm 1 Optimal Multiple Arm Identification (Opt MAI); Algorithm 2 Quartile-Elimination(QE); Algorithm 3 Accept-Reject(AR). |
| Open Source Code | No | The paper does not contain any statement or link indicating that the source code for the described methodology is publicly available. |
| Open Datasets | Yes | We generate θ from a real recognizing textual entailment (RTE) dataset (Section 4.3 in (Snow et al., 2008)). |
| Dataset Splits | No | The paper describes a multi-armed bandit setup, which involves sequential sampling rather than pre-defined train/validation/test dataset splits typical in supervised learning. |
| Hardware Specification | No | The paper does not specify any hardware (e.g., GPU, CPU models, memory) used for running the experiments. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers (e.g., Python, PyTorch, or other libraries). |
| Experiment Setup | Yes | First, observe that in Opt MAI, Q is an upper bound of the number of samples; while (1 βR)Q < Q is the actual number of samples used, where R is the total number of rounds run by the algorithm. ... Third, in each round of Opt MAI, the ratio of the number of samples between two consecutive rounds is set to be β = e0.2 0.75 0.91. In the real implementation, one could treat this quantity as a tuning parameter to make the algorithm more flexible (as long as β (0.75, 1)). In this experiment, we report the results for both β = 0.8 and β = 0.9. |