Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction
Authors: Jacob Steinhardt, Gregory Valiant, Moses Charikar
NeurIPS 2016 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | Our technical tools draw on semidefinite programming methods for matrix completion... The proof of Theorem 2 can be split into two parts: analyzing Algorithm 1 (Section 4), and analyzing Algorithm 2 (Section 5). Proving Proposition 1 involves two major steps... Lemma 1. Let m n. Suppose that Assumption 1 holds. Lemma 2. Let m n. Then there is a k = O log3(2/δ) n such that... Lemma 3. Let i be the row selected in Algorithm 2. Suppose that r satisfies Assumption 1. Lemma 4. The output T of Algorithm 4 satisfies E[T] = T0, T 0 βm. |
| Researcher Affiliation | Academia | Jacob Steinhardt Stanford University Gregory Valiant Stanford University Moses Charikar Stanford University |
| Pseudocode | Yes | Algorithm 1 Algorithm for recovering β-quantile matrix M using (unreliable) ratings A. Algorithm 2 Algorithm for recovering an accurate β-quantile T from the β-quantile matrix M. Algorithm 3 Algorithm for obtaining (unreliable) ratings matrix A and noisy ratings r, r . Algorithm 4 Randomized rounding algorithm. |
| Open Source Code | No | No statement regarding the release of open-source code for the described methodology was found. No links to repositories or mentions of supplementary materials containing code were present. |
| Open Datasets | No | The paper is theoretical and describes a model with 'n workers' and 'm items' but does not specify or use any particular dataset for empirical evaluation, nor does it provide access information for any dataset. |
| Dataset Splits | No | This paper is theoretical and does not involve empirical data or experiments, therefore there is no mention of dataset splits for training, validation, or testing. |
| Hardware Specification | No | This paper is theoretical and does not describe any empirical experiments, thus no hardware specifications for running experiments are mentioned. |
| Software Dependencies | No | This paper is theoretical and does not describe any empirical experiments or implementations, thus no software dependencies with version numbers are mentioned. |
| Experiment Setup | No | This paper is theoretical and does not describe any empirical experiments, thus no experimental setup details like hyperparameters or training settings are provided. |