Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Streaming Bayesian Inference for Crowdsourced Classification
Authors: Edoardo Manino, Long Tran-Thanh, Nicholas Jennings
NeurIPS 2019 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In Section 4 we compute its asymptotical accuracy. In Section 5 we compare its performance with the state of the art on both synthetic and real-world datasets. |
| Researcher Affiliation | Academia | Edoardo Manino University of Southampton EMAIL Long Tran-Thanh University of Southampton EMAIL Nicholas R. Jennings Imperial College, London EMAIL |
| Pseudocode | Yes | Algorithm 1 Fast SBIC Input: dataset X, availability a, policy π, prior θ Output: final predictions ˆy T ... Algorithm 2 Sorted SBIC Input: dataset X, availability a, policy π, prior θ Output: final predictions ˆy T |
| Open Source Code | No | The paper does not provide a direct link to open-source code for the SBIC algorithm or explicitly state that the code is publicly available. |
| Open Datasets | Yes | Second, we consider the 5 publicly available dataset listed in Table 1, which come with binary annotations and ground-truth values. For more information on the datasets see [Snow et al., 2008; Welinder et al., 2010; Lease and Kazai, 2011]. |
| Dataset Splits | No | The paper discusses synthetic and real-world datasets and analyzes prediction error, but it does not explicitly provide details about training, validation, or test dataset splits (e.g., percentages or counts). |
| Hardware Specification | No | The authors acknowledge the use of the IRIDIS High Performance Computing Facility, and associated support services at the University of Southampton. This is a general facility name, but no specific hardware components (e.g., GPU/CPU models, memory) are detailed. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers (e.g., Python 3.x, PyTorch 1.x) that would be needed to reproduce the experiments. |
| Experiment Setup | Yes | To do so, we extract workers from a distribution pj Beta(4, 3), representing a non-uniform population with large variance. ... Additionally, we set the number of tasks to M = 1000 and the number of labels per worker to L = 10. ... we run EM, AMF, MC and SBIC with parameters α and β matching the distribution of pj. ... we run EM, AFM, MC and SBIC with the generic prior α = 2, β = 1 and q = 1/2 as proposed in Liu et al. [2012]. |