Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference
Authors: Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, Zirui Liu
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate that the reproducibility of LLM performance is fragile: changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses. [...] Through carefully controlled experiments across various hardware, software, and precision settings, we quantify when and how model outputs diverge. |
| Researcher Affiliation | Collaboration | 1Rice University 2University of Minnesota Twin Cities 3Adobe Inc. EMAIL EMAIL EMAIL |
| Pseudocode | No | The paper describes the 'Layer Cast' method and its workflow in Figure 7, which is a diagram outlining steps within a transformer block. However, it does not present a formal pseudocode block or algorithm. |
| Open Source Code | Yes | Code is available at https://github.com/nanomaoli/llm_reproducibility. |
| Open Datasets | Yes | We conduct experiments on four recent LLMs, including two reasoning models: Deep Seek-R1-Distill Qwen-7B, Deep Seek-R1-Distill-Llama-8B and two non-reasoning models: Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct [13, 34, 22], across five commonly used LLM evaluation benchmarks: AIME 24, MATH500, Live Code Bench-Easy, Live Code Bench-Medium, and Live Code Bench-Hard [1, 15, 18]. |
| Dataset Splits | No | The paper mentions using evaluation benchmarks (AIME 24, MATH500, Live Code Bench) and the number of independent runs for random sampling (16 and 64 for AIME 24, 4 for MATH500), but it does not specify how these datasets are split into training, validation, or test sets for their experiments, or if they relied on predefined splits without explicit description. |
| Hardware Specification | Yes | For each model-task pair, we evaluate under 12 different runtime configurations, representing all combinations of 2 GPU types (NVIDIA L40S and A100), 2 GPU counts (2 and 4), and 3 batch sizes (8, 16, and 32), i.e., 2 × 2 × 3 = 12 different configurations, to simulate the diversity of deployment environments encountered in real-world evaluations. |
| Software Dependencies | No | The paper mentions using v LLM [20] as the inference backend and Hugging Face Transformers for verification experiments, and also refers to Py Torch. However, it does not provide specific version numbers for any of these software dependencies. |
| Experiment Setup | Yes | For reasoning models, we set the maximum output token length to 32,768 and for non-reasoning models, we set it to 2,048. We primarily use v LLM [20] as the inference backend; to verify that our findings are not backend-specific, we also conduct verification experiments using Hugging Face Transformers (see Appendix for details). In the random sampling experiments, we set temperature to 0.7 and top-p to 0.95. Our Evaluation implementation and prompt setting is adapted from Sky Thought-evals [29], more details can be found in Appendix C. For each model-task pair, we evaluate under 12 different runtime configurations, representing all combinations of 2 GPU types (NVIDIA L40S and A100), 2 GPU counts (2 and 4), and 3 batch sizes (8, 16, and 32). |