Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Position: LLM Social Simulations Are a Promising Research Method
Authors: Jacy Reese Anthis, Ryan Liu, Sean M Richardson, Austin C. Kozlowski, Bernard Koch, Erik Brynjolfsson, James Evans, Michael S. Bernstein
ICML 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | In this position paper, we argue that the promise of LLM social simulations can be achieved by addressing five tractable challenges. We ground our argument in a review of empirical comparisons between LLMs and human research subjects, commentaries on the topic, and related work. Our argument is grounded in a literature review of empirical studies that have compared human research subjects to LLMs, commentaries on the topic, and related work in social science and other LLM applications. |
| Researcher Affiliation | Academia | 1University of Chicago 2Stanford University 3Sentience Institute 4Princeton University 5Santa Fe Institute. Correspondence to: Jacy Reese Anthis <EMAIL>. |
| Pseudocode | No | The paper does not contain any clearly labeled pseudocode or algorithm blocks. The methodologies discussed are descriptive and refer to methods used in other studies. |
| Open Source Code | No | This paper is a position paper and a literature review, it does not present a novel methodology with associated source code. Therefore, it does not provide concrete access to source code for its own methodology. |
| Open Datasets | No | The paper is a literature review that discusses various datasets used in other research (e.g., Moral Foundations Questionnaire-2, American National Election Studies, General Social Survey in Table A1). However, it does not provide concrete access information (links, DOIs, repositories, or direct citations with authors/year for dataset access) for a dataset used in its own work or created by the authors of this paper. |
| Dataset Splits | No | This paper is a position paper and literature review; it does not present new experimental work or utilize specific dataset splits for its own analysis. Therefore, it does not provide specific dataset split information. |
| Hardware Specification | No | The paper does not describe any experiments conducted by the authors. Thus, it does not provide specific hardware details (GPU/CPU models, processor types, or memory amounts) used for running experiments. |
| Software Dependencies | No | The paper does not describe any specific software implementation of a methodology, and therefore does not list software dependencies with version numbers. |
| Experiment Setup | No | The paper is a position paper and literature review, not a report of new experimental work. Therefore, it does not provide details about an experimental setup, hyperparameters, or system-level training settings. |