Measuring Performance of Peer Prediction Mechanisms Using Replicator Dynamics
Authors: Victor Shnayder, Rafael M. Frongillo, David C. Parkes
IJCAI 2016 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We study different mechanism designs, using models estimated from real peer evaluations in several massive online courses. Among other observations, we confirm that recent mechanisms present a significant improvement in robustness over earlier approaches. |
| Researcher Affiliation | Academia | Victor Shnayder ed X; Harvard SEAS shnayder@seas.harvard.edu Rafael M. Frongillo CU Boulder raf@colorado.edu David C. Parkes Harvard SEAS parkes@eecs.harvard.edu |
| Pseudocode | No | The paper describes the peer prediction mechanisms and strategies in narrative text and mathematical equations, but it does not include any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any statement about releasing source code for the methodology described, nor does it include a link to a code repository. |
| Open Datasets | No | The paper mentions using '325,523 peer assessments from 17 courses from a major MOOC platform' and states 'We base our model fit on student reports, not the unobservable signals, which are not available. For the current work, in the absence of better data sets, we will simply stipulate that these are representative of true world models.' However, it does not provide concrete access information (link, DOI, repository, or formal citation with author/year for a publicly available dataset) for this data. |
| Dataset Splits | No | The paper does not specify traditional training/validation/test dataset splits. It describes using 'world models' and '100 starting strategy profiles uniformly at random in the strategy simplex' for simulation, but this is not equivalent to specifying data partitioning for model training and validation. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used to run the experiments, such as GPU/CPU models, memory, or cloud instance specifications. |
| Software Dependencies | No | The paper does not list any specific software dependencies with version numbers (e.g., Python 3.x, PyTorch 1.x, or specific solvers like CPLEX 12.x). |
| Experiment Setup | No | While the paper describes the process of setting up simulations (e.g., 'numerically solve this equation', 'choose 100 starting strategy profiles uniformly at random'), it does not provide specific experimental setup details such as hyperparameters, learning rates, batch sizes, or optimizer settings, which are typically found in empirical research. |