Impartial Peer Review
Authors: David Kurokawa, Omer Lev, Jamie Morgenstern, Ariel D. Procaccia
IJCAI 2015 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | We design an impartial mechanism that selects a k-subset of proposals that is nearly as highly rated as the one selected by the non-impartial (abstract version of) the NSF pilot mechanism, even when the latter mechanism has the unfair advantage of eliciting honest reviews. In this paper, we alleviate these concerns by proposing a peer review mechanism which is not susceptible to such manipulations. Each PI who submits a proposal or paper will review some other PIs proposals or papers. Our mechanism is impartial: reviewers will not be able to affect the chances of their own proposals being selected. Our research challenge is therefore to design provably impartial peer review mechanisms that provide formal quality guarantees. |
| Researcher Affiliation | Academia | David Kurokawa Carnegie Mellon dkurokaw@cs.cmu.edu Omer Lev Hebrew University omerl@cs.huji.ac.il Jamie Morgenstern Carnegie Mellon jamiemmt@cs.cmu.edu Ariel D. Procaccia Carnegie Mellon arielpro@cs.cmu.edu |
| Pseudocode | Yes | The Credible Subset mechanism, denoted Mcs, formally works as follows. CREDIBLE SUBSET(G, m, k) 1. Draw Gm Gm. 2. P {i / topk(Gm) | if i reported j : w(i, j) = 0, i would be in topk(Gm)} 3. S topk(Gm) P. 4. With probability |S| k+m return a random k-subset of S, and with probability 1 |S| k+m return . |
| Open Source Code | No | The paper does not provide concrete access to source code (e.g., a specific repository link or an explicit code release statement) for the methodology described. |
| Open Datasets | No | The paper does not provide concrete access information for a publicly available or open dataset. This is a theoretical paper focusing on mechanism design. |
| Dataset Splits | No | The paper does not provide specific dataset split information, as it is a theoretical paper and does not conduct experiments with empirical data. |
| Hardware Specification | No | The paper does not provide specific hardware details used for running its experiments. This is a theoretical paper. |
| Software Dependencies | No | The paper does not provide specific ancillary software details (e.g., library or solver names with version numbers). This is a theoretical paper. |
| Experiment Setup | No | The paper does not provide specific experimental setup details (e.g., hyperparameter values, training configurations, or system-level settings). This is a theoretical paper that focuses on mechanism design and mathematical analysis. |