Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Impartial Peer Review
Authors: David Kurokawa, Omer Lev, Jamie Morgenstern, Ariel D. Procaccia
IJCAI 2015 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | We design an impartial mechanism that selects a k-subset of proposals that is nearly as highly rated as the one selected by the non-impartial (abstract version of) the NSF pilot mechanism, even when the latter mechanism has the unfair advantage of eliciting honest reviews. In this paper, we alleviate these concerns by proposing a peer review mechanism which is not susceptible to such manipulations. Each PI who submits a proposal or paper will review some other PIs proposals or papers. Our mechanism is impartial: reviewers will not be able to affect the chances of their own proposals being selected. Our research challenge is therefore to design provably impartial peer review mechanisms that provide formal quality guarantees. |
| Researcher Affiliation | Academia | David Kurokawa Carnegie Mellon EMAIL Omer Lev Hebrew University EMAIL Jamie Morgenstern Carnegie Mellon EMAIL Ariel D. Procaccia Carnegie Mellon EMAIL |
| Pseudocode | Yes | The Credible Subset mechanism, denoted Mcs, formally works as follows. CREDIBLE SUBSET(G, m, k) 1. Draw Gm Gm. 2. P {i / topk(Gm) | if i reported j : w(i, j) = 0, i would be in topk(Gm)} 3. S topk(Gm) P. 4. With probability |S| k+m return a random k-subset of S, and with probability 1 |S| k+m return . |
| Open Source Code | No | The paper does not provide concrete access to source code (e.g., a specific repository link or an explicit code release statement) for the methodology described. |
| Open Datasets | No | The paper does not provide concrete access information for a publicly available or open dataset. This is a theoretical paper focusing on mechanism design. |
| Dataset Splits | No | The paper does not provide specific dataset split information, as it is a theoretical paper and does not conduct experiments with empirical data. |
| Hardware Specification | No | The paper does not provide specific hardware details used for running its experiments. This is a theoretical paper. |
| Software Dependencies | No | The paper does not provide specific ancillary software details (e.g., library or solver names with version numbers). This is a theoretical paper. |
| Experiment Setup | No | The paper does not provide specific experimental setup details (e.g., hyperparameter values, training configurations, or system-level settings). This is a theoretical paper that focuses on mechanism design and mathematical analysis. |