Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Incentivizing Truthful Language Models via Peer Elicitation Games
Authors: Baiting Chen, Tong Zhu, Jiale Han, Lexin Li, Gang Li, Xiaowu Dai
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirical evaluations across multiple benchmarks demonstrate significant improvements in factual accuracy. These results position PEG as a practical approach for eliciting truthful behavior from LLMs without supervision or fine-tuning. |
| Researcher Affiliation | Academia | Baiting Chen UCLA EMAIL Tong Zhu UCLA EMAIL Jiale Han UCLA EMAIL Lexin Li UC Berkeley EMAIL Gang Li UCLA EMAIL Xiaowu Dai UCLA EMAIL |
| Pseudocode | Yes | B PEG Algorithm Algorithm 1 Two-Phase PEG Algorithm with Task Batches |
| Open Source Code | Yes | Code for all experiments is available at https: //github.com/toz015/neurips2025-repo. |
| Open Datasets | Yes | We conduct our evaluations using four diverse datasets: ARC-Easy, ARC-Challenge, Massive Multitask Language Understanding (MMLU) and Graduate-Level Google-Proof Q&A Benchmark (GPQA) [21, 14, 48]. |
| Dataset Splits | No | The paper mentions using well-known benchmark datasets but does not explicitly provide specific details about training/test/validation splits. It mentions |
| Hardware Specification | Yes | GPU: A single NVIDIA A100 GPU with 200 GB of memory. CPU: AMD EPYC 7542 or 9654 processors. GPU: NVIDIA RTX 2080 Ti. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers (e.g., Python 3.8, PyTorch 1.9) required to replicate the experiments. |
| Experiment Setup | Yes | Unless otherwise specified, we set the learning rate η = 0.1 for all experiments. The PEG mechanism between discriminators is run for 10 iterations for 8 tasks. Discussions on different choices of learning rates and number of iterations are provided in Appendix C.6 and C.7. |