Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation
Authors: Xinyu Yang, Yuwei An, Hongyi Liu, Tianqi Chen, Beidi Chen
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | After a 3-hour fine-tuning with 1K examples, our Multiverse-32B stands as the only opensourced non-AR model achieving performance on par with leading AR-LLMs of the same scale, evidenced by AIME24 & 25 scores of 54% and 46%, respectively. Moreover, our budget control experiments show that Multiverse-32B exhibits superior scaling, outperforming AR-LLMs by 1.87% on average using the same context length. |
| Researcher Affiliation | Collaboration | Xinyu Yang , Yuwei An , Hongyi Liu , Tianqi Chen , Beidi Chen CMU, Nvidia xinyuya2, yuweia, hongyil2, tqchen, EMAIL |
| Pseudocode | No | The paper describes the |
| Open Source Code | Yes | We have open-sourced the entire Multiverse ecosystem, including data, model weights, serving system, supporting tools, as well as data curation prompts and detailed training and evaluation recipes. |
| Open Datasets | Yes | In practice, this process results in Multiverse-1K, a dataset of 1,000 high-quality structured training samples for advancing LLM reasoning. ... We evaluate the effectiveness and efficiency of Multiverse in real-world reasoning tasks. Specifically, ... Multiverse-32B on four tasks, including AIME24 [24], AIME25 [25], MATH500 [18], and GPQA Diamond [35]. |
| Dataset Splits | No | The training data consisted of a combination of Multiverse 1K prompted with Think step by step and in parallel , and the original sequential data appended by Think step by step , using a mixture ratio increased from 0:1 (all original data) to 1:0 (all our data) across eight epochs. While the paper mentions the training data composition and evaluation benchmarks, it does not explicitly provide specific percentages, sample counts, or methodology for dataset splits into training, validation, or test sets. |
| Hardware Specification | Yes | Our fine-tuning took 3 hours on 8 NVIDIA B200 GPUs with Py Torch FSDP. |
| Software Dependencies | No | Our fine-tuning took 3 hours on 8 NVIDIA B200 GPUs with Py Torch FSDP. Specifically, we employ SGLang [50] due to its support for continuous batching and radix attention. The paper mentions PyTorch and SGLang but does not provide specific version numbers for these software components. |
| Experiment Setup | Yes | Our fine-tuning took 3 hours on 8 NVIDIA B200 GPUs with Py Torch FSDP. The training data consisted of a combination of Multiverse 1K prompted with Think step by step and in parallel , and the original sequential data appended by Think step by step , using a mixture ratio increased from 0:1 (all original data) to 1:0 (all our data) across eight epochs. Empirically, Multiverse-32B achieves significant performance improvement, outperforming the base model by 23.6%, with AIME24 and AIME25 scores of 53.8% and 45.8%, respectively. We test our model under two prompting conditions: with and without the phrase in parallel . |