Reputation-based Worker Filtering in Crowdsourcing
Authors: Srikanth Jagabathula, Lakshminarayanan Subramanian, Ashwin Venkataraman
NeurIPS 2014 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the effectiveness of our algorithm using a combination of strong theoretical guarantees and empirical results on real-world datasets. |
| Researcher Affiliation | Academia | 1Department of IOMS, NYU Stern School of Business 2Department of Computer Science, New York University 3CTED, New York University Abu Dhabi |
| Pseudocode | Yes | Specifically, we propose 2 computationally efficient algorithms to compute worker reputations using only the labels provided by the workers (see Algorithms 1 and 2), which are robust to manipulation by adversaries. |
| Open Source Code | No | The paper does not provide any statement or link indicating that its source code is publicly available. |
| Open Datasets | Yes | Finally, using several publicly available crowdsourcing datasets (see Section 4), we show that our reputation algorithm: (a) can help in enhancing the accuracy of state-of-the-art label aggregation algorithms (b) is able to detect workers in these datasets who exhibit certain non-random strategies. |
| Dataset Splits | No | The paper mentions using synthetic and real datasets but does not specify any training, validation, or test split percentages or exact counts for its experiments. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used to run the experiments. |
| Software Dependencies | No | The paper mentions using popular label aggregation algorithms (MV, EM, KOS, KOS+) and implementing iterative versions of its own algorithms, but it does not specify any software names with version numbers. |
| Experiment Setup | Yes | We simulated a setup of 100 workers with a power-law distribution for worker degrees to generate the bipartite worker-task assignment graph. We assume that an honest worker always labels correctly (the results are qualitatively similar when honest workers make errors with small probability) and consider three notions of adversaries: (a) random who label each task 1 or 1 with prob. 1/2 (b) malicious who always label incorrectly and (c) uniform who label 1 on all tasks. Also, we consider both cases one where the adversaries are biased to have high degrees and the other where they have low degrees. Further, we arbitrarily decided to remove 15% of the workers with the highest penalties and we define precision as the percentage of workers filtered who were adversarial. |