A Market-Inspired Bidding Scheme for Peer Review Paper Assignment
Authors: Reshef Meir, Jérôme Lang, Julien Lesca, Nicholas Mattei, Natan Kaminsky4776-4784
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show via extensive simulations on bidding data from real conferences, that our bidding scheme would substantially improve both the bid distribution and the resulting assignment. and In-Silico Experiments In order to test the effect of the TP bidding scheme, we simulated PCMs who interact with a bidding system. The PCMs observe dynamic paper iprices and bid in turn. To keep simulations as realistic as possible, we used bidding data from real conferences to generate PCMs costs and behaviors. |
| Researcher Affiliation | Academia | 1Technion Israel Institute of Technology 2Universit e Paris Dauphine 3 Tulane University |
| Pseudocode | No | The paper describes algorithms conceptually and references a more detailed version in an appendix, but does not contain structured pseudocode or algorithm blocks in the main text. |
| Open Source Code | No | The paper states: We defer most proofs and simulation results to the appendix of the full version at https://tinyurl.com/y8z4y3z4. This link is to a PDF document, not source code. The paper also mentions using implementations from other works (e.g., implementations from Aziz et al. (2019b)) and publicly available datasets (Pref Lib), but it does not provide concrete access to its own source code for the described methodology. |
| Open Datasets | Yes | Datasets We used all 5 bidding datasets (DP1-DP5) available on Pref Lib (Mattei and Walsh 2013, 2017). In addition, we used random samples in varying proportions from another large AI conference (DA1-DA3). ... We also used two datasets (DI1, DI2) sampled from the ICLR 18 dataset used in Fiez, Shah, and Ratliff (2019). |
| Dataset Splits | No | The paper describes simulating bidding behavior on full datasets obtained from real conferences (e.g., Pref Lib, ICLR 18 dataset). It does not describe explicit train/validation/test splits of data for model training or evaluation in the conventional sense, as it focuses on evaluating a bidding scheme through simulation. |
| Hardware Specification | No | The paper describes conducting extensive simulations but does not provide any specific hardware details such as GPU/CPU models or other system specifications used for these simulations. |
| Software Dependencies | No | The paper mentions using algorithms (Utilitarian, Egalitarian, Rank Maximal) and existing implementations (e.g., implementations from Aziz et al. (2019b)), but it does not specify any software names with version numbers for dependencies or the simulation environment. |
| Experiment Setup | Yes | In the trading post bidding scheme, we let a PCM bid exactly once, in random order. ... every PCM starts with a virtual bid of k/m on each paper... We update the iprices every 5 bids. ... The PCM bids on papers in increasing order of (Cij − β pj), until their cumulative iprice reaches or exceeds R. Unless specified otherwise, we use β = 2. ... We use r = 3. ... We generated costs in the ranges [0, 1] for strong bids, [1, 2] for weak bids, and [2, 8] for no bid... |