PeerNomination: Relaxing Exactness for Increased Accuracy in Peer Selection
Authors: Nicholas Mattei, Paolo Turrini, Stanislav Zhydkov
IJCAI 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show empirically that it achieves higher accuracy than the exiting algorithms over several metrics. ... We extend the testing framework developed by Aziz et al. [2019] and using methods from PREFLIB [Mattei and Walsh, 2017]. Our code and data is available online 1. The experiment was repeated 1000 times for each setting, after which the average recall was calculated giving us high confidence in our results. |
| Researcher Affiliation | Academia | 1Department of Computer Science, Tulane University 2Department of Computer Science, University of Warwick nsmattei@tulane.edu, {p.turrini, s.zhydkov}@warwick.ac.uk |
| Pseudocode | Yes | Algorithm 1 PEERNOMINATION |
| Open Source Code | Yes | Our code and data is available online 1. 1https://github.com/nmattei/peerselection |
| Open Datasets | No | The paper states: 'In a Mallows model we provide a (random) ground truth ranking π and a noise parameter φ.' and 'For each setting of the parameters we generated a random m-regular assignment matching reviewers to reviewees.' This indicates that data was generated based on a model rather than using a pre-existing publicly available dataset with specific access details. |
| Dataset Splits | No | The paper does not specify explicit training, validation, or test dataset splits. It describes generating data via a Mallows model and repeating experiments 1000 times, but no fixed data partitioning is mentioned. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware (e.g., GPU models, CPU types, memory) used to run the experiments. |
| Software Dependencies | No | The paper mentions extending a testing framework and using methods from PREFLIB but does not list any specific software dependencies with version numbers (e.g., Python, PyTorch, or specific solvers with versions). |
| Experiment Setup | Yes | As in Aziz et al. [2019], we set n = 120 and tested the algorithm on various values of k and m. The test values for k were 15, 20, 25, 30, 35 and the test values for m were 5, 7, 9, 11. For the algorithms that rely on the partition, we chose the number of partitions, l, to be 4. For each setting of the parameters we generated a random m-regular assignment matching reviewers to reviewees. ... In a Mallows model we provide a (random) ground truth ranking π and a noise parameter φ. ... The experiment was repeated 1000 times for each setting, after which the average recall was calculated giving us high confidence in our results. For PEERNOMINATION, we used theoretical estimates of ε to achieve the right expected size of the accepting set. |