Achieving budget-optimality with adaptive schemes in crowdsourcing
Authors: Ashish Khetan, Sewoong Oh
NeurIPS 2016 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our approach is different from existing adaptive schemes in [5], where there are multiple types of tasks and the main source of uncertainty is which type the next arriving worker is expert on. Golden tasks with known answers are used to explore expertise and tasks are assigned accordingly. ... numerical simulations confirm the superiority compared to state-of-the-art non-adaptive schemes. ... In Figure 1, we compare performance of our algorithm with majority voting and also non-adaptive version of our Algorithm 1, where we assign to each task ℓ(the given budget) number of workers in one round and set classification threshold Xt,u = 0 so as to classify all the tasks. ... We run synthetic experiments with m = 1800 and fix n = 1800 for the non-adaptive version. The crowds are generated from the spammer-hammer model with hammer probability equal to 0.3. In the left panel, we take difficulty level λa to be uniformly distributed over {1, 1/4, 1/16}, that gives λ = 1/7. In the right panel, we take λa = 1 with probability 3/4, otherwise we take it to be 1/4 or 1/16 with equal probability, that gives λ = 4/13. As predicted from the theoretical analysis, our adaptive algorithm improves significantly over its non-adaptive version. ... We run synthetic experiments with m = n = 1000 and the crowds are generated from the spammer-hammer model where pj = 1 with probability β and 1/2 otherwise. We fix β = 0.3 and vary ℓin the left figure and fix ℓ= 30 and vary β in the right figure. |
| Researcher Affiliation | Academia | Ashish Khetan and Sewoong Oh Department of ISE, University of Illinois at Urbana-Champaign Email: {khetan2,swoh}@illinois.edu |
| Pseudocode | Yes | Algorithm 1 Adaptive Task Assignment and Inference Algorithm ... Algorithm 2 Message-Passing Algorithm |
| Open Source Code | No | The paper does not provide any specific link or statement indicating that the source code for the described methodology is publicly available. |
| Open Datasets | No | We run synthetic experiments with m = 1800 and fix n = 1800 for the non-adaptive version. The crowds are generated from the spammer-hammer model with hammer probability equal to 0.3. In the left panel, we take difficulty level λa to be uniformly distributed over {1, 1/4, 1/16}, that gives λ = 1/7. In the right panel, we take λa = 1 with probability 3/4, otherwise we take it to be 1/4 or 1/16 with equal probability, that gives λ = 4/13. ... We run synthetic experiments with m = n = 1000 and the crowds are generated from the spammer-hammer model where pj = 1 with probability β and 1/2 otherwise. We fix β = 0.3 and vary ℓin the left figure and fix ℓ= 30 and vary β in the right figure. |
| Dataset Splits | No | The paper describes generating synthetic data and conducting experiments, but it does not specify explicit training, validation, or test dataset splits with percentages, sample counts, or references to predefined splits. |
| Hardware Specification | No | The paper does not provide any specific hardware details (e.g., CPU, GPU models, memory, or cloud instances) used for running the experiments. |
| Software Dependencies | No | The paper does not specify any software names with version numbers for reproducibility. |
| Experiment Setup | Yes | We run synthetic experiments with m = 1800 and fix n = 1800 for the non-adaptive version. The crowds are generated from the spammer-hammer model with hammer probability equal to 0.3. In the left panel, we take difficulty level λa to be uniformly distributed over {1, 1/4, 1/16}, that gives λ = 1/7. In the right panel, we take λa = 1 with probability 3/4, otherwise we take it to be 1/4 or 1/16 with equal probability, that gives λ = 4/13. ... For a fair comparison with the non-adaptive version, we fix total budget to be mℓand assign workers in each round until the budget is exhausted. Cδ is 1 and st = 1 for t {1, 2, 3}. ... We run synthetic experiments with m = n = 1000 and the crowds are generated from the spammer-hammer model where pj = 1 with probability β and 1/2 otherwise. We fix β = 0.3 and vary ℓin the left figure and fix ℓ= 30 and vary β in the right figure. We let qi s take values in {0.6, 0.8, 1} with equal probability such that α = 1.4/3. |