Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Integrating Rankings into Quantized Scores in Peer Review
Authors: Yusha Liu, Yichong Xu, Nihar B Shah, Aarti Singh
TMLR 2022 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically evaluate our method on synthetic datasets as well as on peer reviews from the ICLR 2017 conference, and find that it reduces the error by approximately 30% as compared to the best performing baseline on the ICLR 2017 data. |
| Researcher Affiliation | Collaboration | Yusha Liu EMAIL Machine Learning Department Carnegie Mellon University Yichong Xu EMAIL Microsoft Cognitive Services Research Nihar B. Shah EMAIL Machine Learning Department, Computer Science Department Carnegie Mellon University Aarti Singh EMAIL Machine Learning Department Carnegie Mellon University |
| Pseudocode | Yes | Algorithm 1 Proposed algorithm Algorithm 2 Quantization Validation (QV) Algorithm 3 BRE-adjusted-scores Algorithm 4 Partial-rankings-adjusted-scores |
| Open Source Code | Yes | The code for our algorithms and results is available online. https://github.com/MYusha/rankings_and_quantized_scores |
| Open Datasets | Yes | We conduct experiments on data from the peer-review process of the ICLR 2017 conference (Kang et al., 2018). |
| Dataset Splits | No | The paper describes generating a 'synthetic dataset' and a 'semi-synthetic dataset based on real data from ICLR 2017', but does not specify training/test/validation splits in the conventional sense for reproducing experiments. |
| Hardware Specification | No | The paper does not explicitly mention any specific hardware used for running the experiments (e.g., GPU models, CPU types, or cloud resources with specifications). |
| Software Dependencies | No | We use CVXPY to obtain the solutions: The solver we used is CVXOPT with the tolerance for feasibility conditions (feastol) set as 10-6. (No specific versions mentioned for CVXPY or CVXOPT) |
| Experiment Setup | Yes | The constant ฯต which enforces the strict inequality constraints is set as 0.05. ... We use an exponential grid for candidate values of ฮป in Quantization Validation: ฮ = {exp(t/4) : 0 โค t < 40, t โ Z}. The quantization function is set to q(ยท) = โยท/2โ. |