Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Peer Prediction for Learning Agents
Authors: Shi Feng, Fang-Yi Yu, Yiling Chen
NeurIPS 2022 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Simulation of several algorithms in this family as well as the ϵ-greedy algorithm, which is outside of this family, shows convergence to the truthful strategy in the CA mechanism. [...] We simulate the CA mechanism with various learning algorithms: the Hedge algorithms, follow the perturbed leader, follow the leader, and ϵ-greedy, and repeat the process 400 times with 800 rounds on each algorithm each time. |
| Researcher Affiliation | Academia | Shi Feng Institute for Interdisciplinary Information Sciences, Tsinghua University Beijing, China [...] Fang-Yi Yu Department of Computer Science, George Mason University Fairfax, VA, USA [...] Yiling Chen John A. Paulson School of Engineering and Applied Sciences, Harvard University Cambridge, MA, USA |
| Pseudocode | No | The paper describes algorithms and processes mathematically and textually, but it does not include any formal pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any statement about releasing source code or a link to a code repository for the methodology described. |
| Open Datasets | No | The paper describes a simulation setup for signal distribution ("PX,Y (0, 0) = PX,Y (1, 1) = 0.4, PX,Y (1, 0) = PX,Y (0, 1) = 0.2") but does not use or provide access information for a publicly available or open dataset. |
| Dataset Splits | No | The paper conducts simulations but does not describe the use of dataset splits for training, validation, or testing in the conventional machine learning sense. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware (e.g., GPU/CPU models, memory) used to run the simulations. |
| Software Dependencies | No | The paper discusses various learning algorithms (Hedge, FPL, FTL, ϵ-greedy) but does not specify software dependencies or their version numbers used in the simulations. |
| Experiment Setup | Yes | We simulate the CA mechanism with various learning algorithms: the Hedge algorithms, follow the perturbed leader, follow the leader, and ϵ-greedy, and repeat the process 400 times with 800 rounds on each algorithm each time. [...] In our simulations, we use the following private signal distribution that satisfies assumption 2.2: PX,Y (0, 0) = PX,Y (1, 1) = 0.4, PX,Y (1, 0) = PX,Y (0, 1) = 0.2. [...] FPL algorithm given a noise distribution N on scalars, f FPL i (R1, R2, R3, R4) = Pr n Ri + pi = maxj [4]{Rj + pj} pj iid N, j [4] o for i [4]. We consider FPL1, FPL4, and FPL8. Hedge algorithm 1 choose i with probability proportional to 3Ri/2 that is an implementation by choosing ϵ = 0.5 for the multiplicative weights algorithm of Arora et al. [2]. Hedge algorithm 2 chooses i with probability proportional to e Ri that is an implementation by choosing β = 1 of exponentially weighted averaged forecaster introduced by Freund and Schapire [10]. Finally, ϵ-greedy algorithm uses time varying ϵ = 1 (t+1)2 at round t. |