Approval Voting and Incentives in Crowdsourcing
Authors: Nihar Shah, Dengyong Zhou, Yuval Peres
ICML 2015 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We also conduct preliminary empirical studies on Amazon Mechanical Turk which validate our approach. |
| Researcher Affiliation | Collaboration | Nihar B. Shah NIHAR@EECS.BERKELEY.EDU University of California, Berkeley, CA 94720 Dengyong Zhou DENZHO@MICROSOFT.COM Microsoft Research, Redmond, WA 98052 Yuval Peres PERES@MICROSOFT.COM Microsoft Research, Redmond, WA 98052 |
| Pseudocode | Yes | Algorithm 1 Incentive mechanism for approval voting |
| Open Source Code | No | The paper states 'The entire data related to the experiments is available on the website of the first author' but does not mention code for the methodology. |
| Open Datasets | No | The paper states 'The entire data related to the experiments is available on the website of the first author' but does not provide a specific link, DOI, repository, or formal citation for a publicly available dataset. It refers to data collected by the authors. |
| Dataset Splits | No | The paper does not explicitly provide details about training, validation, or test dataset splits. |
| Hardware Specification | No | The paper mentions Amazon Mechanical Turk as the platform for crowdsourcing but does not provide specific hardware details (e.g., GPU/CPU models, memory) used for running its experiments or models. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers. |
| Experiment Setup | Yes | In each experiment, every worker was assigned one of four mechanisms uniformly at random. The mechanisms were executed as a bonus payment based on the evaluation of the worker s performance on the gold standard questions, on top of a guaranteed payment of 10 cents. The four mechanisms tested were: Single-selection interface with additive payments: The bonus starts at zero and is increased by a fixed amount for every correct answer. Skip-based single-selection interface with multiplicative payments (Shah & Zhou, 2014): The bonus starts at a certain positive value, is reduced by a certain fraction for each skipped question, and becomes zero in case of an incorrect answer. Approval-voting interface with a fixed payment: The bonus is fixed. Approval-voting interface with the payment defined in Algorithm 1. |