Crowdsourcing with Arbitrary Adversaries
Authors: Matthaeus Kleindessner, Pranjal Awasthi
ICML 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments on both synthetic and real data show that our approach clearly outperforms existing methods in the presence of adversaries. and 6. Experiments On both synthetic and real data, we compared our proposed Algorithm 1 to straightforward majority voting for predicting labels (referred to as Maj) and the following methods from the literature: |
| Researcher Affiliation | Academia | 1Department of Computer Science, Rutgers University, Piscataway Township, New Jersey, USA. Correspondence to: Matth aus Kleindessner <matthaeus.kleindessner@rutgers.edu>, Pranjal Awasthi <pranjal.awasthi@rutgers.edu>. |
| Pseudocode | Yes | Algorithm 1 Input: crowdsourced labels stored in A { 1, 0, +1}m n, upper bound 0 < γTR < 1 2 on the error probabilities of n 2 + 2 workers that follow the one-coin model, confidence parameter 0 < δ < 1 Output: estimates (εF l )l [n], (c F jk)j<k, (ˆyi)i [m] of error probabilities, covariances and ground-truth labels |
| Open Source Code | No | The paper states 'Code available on https://github.com/zhangyuc/ Spectral Methods Meet EM.' but this refers to the code for Zhang et al. (2016), a method they compare against, and not their own proposed methodology. |
| Open Datasets | Yes | We performed experiments on six publicly available data sets that are are commonly used in the literature (cf. Snow et al., 2008, Zhang et al., 2016, and Bonald & Combes, 2017). All six data sets come with ground truth labels for each task. |
| Dataset Splits | No | The paper does not explicitly provide training/validation/test dataset splits (e.g., percentages, absolute sample counts, or citations to predefined splits) for its experiments on either synthetic or real data. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware (e.g., GPU models, CPU types, cloud computing resources) used to run the experiments. |
| Software Dependencies | No | The paper states 'We used the Matlab implementation of KOS, S-EM1 and S-EM10 made available by Zhang et al. (2016)' but does not specify version numbers for Matlab or any other software components. |
| Experiment Setup | Yes | We always called Algorithm 1 with parameters γTR = 0.4 and δ = 0.1, which resulted in γ being set to 1 in the execution of the algorithm in all our experiments. |