Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
SpecEM: Training-Free LLM Ensembling via Iterative Drafting, Verification, and Online Feedback
Authors: Bo Lv, Nayu Liu, Chen Tang, Xin Liu, Yue Yu, Ping Luo
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results on five LLM families (ranging from 7B to 72B parameters) and six benchmark datasets, spanning open-domain instruction following, reasoning, commonsense, demonstrate consistent performance improvements compared to state-of-the-art LLM ensemble methods. |
| Researcher Affiliation | Academia | 1Peng Cheng Laboratory 2Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 3University of Chinese Academy of Sciences 4Tianjin Laboratory Autonomous Intelligence Technology and Systems, School of Computer Science and Technology, Tiangong University 5Institute for Advanced Algorithms Research, Shanghai EMAIL |
| Pseudocode | No | The paper describes the methodology in detail in Section 3 and provides an overview in Figure 1, but it does not include a distinct pseudocode or algorithm block. |
| Open Source Code | Yes | Our code is available at https://github.com/lvbotenbest/Spec EM. |
| Open Datasets | Yes | Datasets. We evaluate Spec EM on six datasets that reflect key capabilities of LLMs, including opendomain instruction following, commonsense, and reasoning. Fuse Eval: A multilingual instruction response benchmark we construct by combining Dolly-15k [Conover et al., 2023] and Alpaca GPT4 [Peng et al., 2023] for English, and Human-Value and Ruozb from COIG-CQIA [Bai et al., 2024] for Chinese. IFEval [Zhou et al., 2023]: Evaluates instruction adherence under four granular settings, prompt-strict/loose, instruction-strict/loose. Alpaca Eval 2.0 [Dubois et al., 2024]: Measures alignment with human preferences via GPT-4 based pairwise comparisons against GPT-4 outputs. MMLU (5-shot) [Hendrycks et al., 2021] and ARC-C (5-shot) [Clark et al., 2018]: Multiple-choice benchmarks that test factual knowledge and general commonsense. GSM8K (3-shot) [Cobbe et al., 2021]: Focuses on arithmetic and multi-step reasoning through grade-school math problems. |
| Dataset Splits | Yes | For Fuse Eval, we choose the Dolly15k [Conover et al., 2023] and Alpaca-gpt4 [Peng et al., 2023] datasets for evaluation... we randomly sample portions from each to create a new test set and a development set. |
| Hardware Specification | Yes | For experiments with 7B 9B models, we use A100 GPUs, while larger models (24B 72B) are evaluated on H200 GPUs. |
| Software Dependencies | No | All models are loaded using bfloat16 precision, with do_sample = True, temperature = 0.6, and top_p = 0.9 generation settings. For the Chinese results, we use Jieba to split the text into words before calculating these two scores. The paper mentions software tools like Jieba but does not specify version numbers for any libraries or frameworks used in the implementation. |
| Experiment Setup | Yes | All models are loaded using bfloat16 precision, with do_sample = True, temperature = 0.6, and top_p = 0.9 generation settings. For experiments with 7B 9B models, we use A100 GPUs, while larger models (24B 72B) are evaluated on H200 GPUs. The maximum number of candidate segments is set to L = 10, and the online feedback hyperparameter is set to α = 1. All reported results are averaged over three independent runs to ensure stability. |