On the Efficiency of Data Collection for Crowdsourced Classification
Authors: Edoardo Manino, Long Tran-Thanh, Nicholas R. Jennings
IJCAI 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | For this reason, we run synthetic experiments with M = 10000 tasks and a uniform distribution of workers fp over the interval [0.4, 0.8] to simulate a mixed crowd (similar results can be obtained with different choices of fp). We report the results in Figure 2, where each point is the average of 100 runs and has standard error below 5 10 4. ... In order to rule out this eventuality, we run synthetic experiments with M = 200 tasks, Q = 10 labels per worker and fp Beta(α=4, β =3) to simulate a mixed crowd. Moreover, we use the approximate variational inference method in [Liu et al., 2012] to aggregate the labels, and average the results over 1000 runs, which yields a standard error below 2 10 3. |
| Researcher Affiliation | Academia | Edoardo Manino1, Long Tran-Thanh1 and Nicholas R. Jennings2 1 University of Southampton 2 Imperial College, London |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide concrete access to source code for the methodology described. |
| Open Datasets | No | The paper uses synthetic experiments with simulated worker distributions (uniform or Beta distribution) and does not provide access information for a public dataset. |
| Dataset Splits | No | The paper does not provide specific dataset split information for training, validation, or testing, as it uses synthetic data generated for simulation rather than a pre-existing dataset with defined splits. |
| Hardware Specification | No | The paper mentions 'the IRIDIS High Performance Computing Facility' but does not provide specific hardware details (e.g., GPU/CPU models, memory amounts) used for running its experiments. |
| Software Dependencies | No | The paper refers to algorithms and methods, such as the 'approximate variational inference method proposed by Liu et al. [2012]', but does not provide specific ancillary software details with version numbers (e.g., library or solver names with version numbers). |
| Experiment Setup | Yes | For this reason, we run synthetic experiments with M = 10000 tasks and a uniform distribution of workers fp over the interval [0.4, 0.8] to simulate a mixed crowd (similar results can be obtained with different choices of fp). ... In order to rule out this eventuality, we run synthetic experiments with M = 200 tasks, Q = 10 labels per worker and fp Beta(α=4, β =3) to simulate a mixed crowd. Moreover, we use the approximate variational inference method in [Liu et al., 2012] to aggregate the labels, and average the results over 1000 runs, which yields a standard error below 2 10 3. |