Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning
Authors: Wanjia Zhao, Mert Yuksekgonul, Shirley Wu, James Y Zou
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments demonstrate that SIRIUS significantly enhances multi-agent performance across multiple domains. It improves reasoning and biomedical QA accuracy by 2.86% to 21.88%, while also strengthening agent negotiation in competitive scenarios. Our results show that SIRIUS enhances multi-agent performance while generating reusable data for self-correction and self-play enhancement in the future. |
| Researcher Affiliation | Academia | Wanjia Zhao, Mert Yuksekgonul, Shirley Wu, James Zou Department of Computer Science Stanford University |
| Pseudocode | Yes | The core training procedure is outlined in Algorithm 1. Algorithm 1 SIRIUS |
| Open Source Code | Yes | Correspondence to EMAIL and EMAIL. Code is available here. |
| Open Datasets | Yes | College Physics/Chemistry. These two datasets are constructed by combining questions from Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2020), Graduate-Level Google-Proof Q&A (GPQA) (Rein et al., 2023), and Theorem-Driven Question Answering (Theorem QA) (Chen et al., 2023). Pub Med QA. This is a biomedical question-answering dataset comprising 1000 open-domain questions (Jin et al., 2019) |
| Dataset Splits | Yes | We split the dataset into training and test sets, with the detailed data distribution provided in Appendix D. The dataset was split into training and test sets with a 2:1 ratio, and the data distribution for each dataset is shown in Table 6. |
| Hardware Specification | No | The paper mentions using gpt-3.5-turbo-0125 and gpt-4o-mini-2024-07-18 as the backbone model and Open AI s Fine-tuning API. This indicates the use of OpenAI's infrastructure, but it does not specify the specific hardware (e.g., GPU models, CPU types, memory) used by the authors for their experiments or fine-tuning. |
| Software Dependencies | No | The paper specifies the use of 'gpt-3.5-turbo-0125' and 'gpt-4o-mini-2024-07-18' as backbone models and 'Open AI s Fine-tuning API'. These are specific models and an API, but the paper does not list other ancillary software components with specific version numbers (e.g., Python version, specific libraries like PyTorch or TensorFlow versions, CUDA versions). |
| Experiment Setup | Yes | For a fair comparison, we use gpt-3.5-turbo-0125 and gpt-4o-mini-2024-07-18 as the backbone model, and set the temperature to 0 in all our experiments. We use Open AI s Fine-tuning API for supervised fine-tuning. The core training procedure is outlined in Algorithm 1, which includes 'total number of fine-tuning Iterations T'. |