Calibrating “Cheap Signals” in Peer Review without a Prior
Authors: Yuxuan Lu, Yuqing Kong
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We perform numerical experiments to compare the performance of our Surprisal-based Score and the baseline score in the binary setting (Σ={0(reject),1(accept)}). Recall that the baseline is the proportion of the accept ratings. Here we describe the parameters we select in numerical experiments. |
| Researcher Affiliation | Academia | Yuxuan Lu Center on Frontiers of Computing Studies School of Computer Science Peking University Beijing, China yx_lu@pku.edu.cn Yuqing Kong Center on Frontiers of Computing Studies School of Computer Science Peking University Beijing, China yuqing.kong@pku.edu.cn |
| Pseudocode | No | The paper provides mathematical formulas and descriptions for calculating scores but does not present a clearly labeled 'Pseudocode' or 'Algorithm' block. |
| Open Source Code | No | The paper does not contain any explicit statement about providing open-source code for the methodology or a link to a code repository. |
| Open Datasets | No | The paper describes numerical experiments using prior distributions (e.g., Beta(1,1)) for synthetic data generation, but it does not specify any publicly available datasets, provide links, DOIs, or citations for dataset access. |
| Dataset Splits | No | The paper describes numerical experiments and parameters for data generation but does not provide details on specific training, validation, or test dataset splits (e.g., percentages, sample counts, or citations to predefined splits). |
| Hardware Specification | No | The paper describes numerical experiments but does not provide any specific details regarding the hardware (e.g., CPU, GPU models, memory) used to run these experiments. |
| Software Dependencies | No | The paper describes numerical experiments and theoretical models but does not mention any specific software dependencies with version numbers (e.g., programming languages, libraries, or solvers). |
| Experiment Setup | Yes | Here we describe the parameters we select in numerical experiments. 1. The number of agents n: We perform the experiments in the settings of n=3 and n=5. 2. The prior distribution of states Q: ... Beta(1/2)(most papers quality is either high or low), Beta(1,1)(papers quality is distributed uniformly), and Beta(3,3)(most papers have a medium quality). 3. The bias vector b: ... opposite: Paper A has the positive bias vector b A=[0,1] ... same: Both paper A and B have the same bias vector b A=b B =[0,1]11. This simulates a situation where both papers have positive (negative) cheap signals. |