Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Uncertainty Quantification for LLM-Based Survey Simulations
Authors: Chengpiao Huang, Yuhang Wu, Kaizheng Wang
ICML 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We illustrate our method on real datasets and LLMs. ... Numerical experiments on real datasets verified the coverage guarantees of our approach, and revealed that existing LLMs exhibited higher fidelity in simulating opinions to social problems than in simulating student answers to mathematics questions. |
| Researcher Affiliation | Academia | 1Department of Industrial Engineering and Operations Research, Columbia University 2Decision, Risk, and Operations Division, Columbia Business School 3Data Science Institute, Columbia University. Correspondence to: Kaizheng Wang <EMAIL>. |
| Pseudocode | Yes | Algorithm 1 Simulation Sample Size Selection |
| Open Source Code | Yes | The code and data are available at https://github.com/yw3453/uqllm-survey-simulation. |
| Open Datasets | Yes | The first dataset is the Opinion QA dataset (Santurkar et al., 2023). ... The second dataset is the EEDI dataset created by (He Yueya et al., 2024), which was built upon the Neur IPS 2020 Education Challenge dataset (Wang et al., 2021). |
| Dataset Splits | Yes | We then randomly split D = {(Dj, Dsyn j )}J j=1 into a training set Dtr = {(Dj, Dsyn j )}j Jtr and a testing set Dte = {(Dj, Dsyn j )}j Jte, with |Dtr| : |Dte| = 3 : 2. ... We consider 100 random train-test splits of the questions. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory amounts) used for running its experiments. |
| Software Dependencies | No | The paper lists various LLMs (e.g., GPT-3.5-Turbo, GPT-4o, Claude 3.5 Haiku) but does not provide specific version numbers for software dependencies like programming languages or libraries used for implementation. |
| Experiment Setup | Yes | Hyperparameters. We consider α {0.05 ℓ: ℓ [10]}, c = 2 and γ = 0.5. For the EEDI dataset, we set the simulation budget K = 50 and take M = 1... For the Opinion QA dataset, we set the simulation budget K = 100 and take M = 2... |