Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Chain-of-Retrieval Augmented Generation
Authors: Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, Furu Wei
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results across multiple benchmarks validate the efficacy of Co RAG, particularly in multi-hop question answering tasks, where we observe more than 10 points improvement in EM score compared to strong baselines. On the KILT benchmark, Co RAG establishes a new state-of-the-art performance across a diverse range of knowledge-intensive tasks. |
| Researcher Affiliation | Collaboration | Microsoft Research Renmin University of China |
| Pseudocode | No | The paper does not contain explicitly labeled pseudocode or algorithm blocks. Figure 2 provides an overview diagram, and Section 3 includes mathematical loss functions, but no structured code-like procedures. |
| Open Source Code | Yes | Our code, data and trained models are available at https://github.com/microsoft/LMOps/tree/main/corag. |
| Open Datasets | Yes | We evaluate Co RAG utilizing two sets of benchmarks: (1) a collection of multi-hop QA datasets, including 2Wiki Multihop QA [8], Hotpot QA [39], Bamboogle [28], and Mu Si Que [32]; (2) the KILT benchmark [27], which encompasses a broad spectrum of knowledge-intensive tasks. |
| Dataset Splits | Yes | The compiled multi-hop QA dataset comprises 125k training instances, whereas the KILT benchmark includes 660k instances after sub-sampling. ... Table 6: Statistics of the datasets used for multi-hop QA training. 2Wiki Multihop QA: # Training Samples 15,000, # Validation Samples 12,576. Hotpot QA: # Training Samples 90,447, # Validation Samples 7,405. Mu Si Que: # Training Samples 19,938, # Validation Samples 2,417. |
| Hardware Specification | Yes | All training jobs are conducted using 8 A100 GPUs. |
| Software Dependencies | No | The paper mentions several models and retrievers used (e.g., Llama-3.1-8B-Instruct, E5-large, E5-Mistral, Rank LLa MA re-ranker) but does not provide specific version numbers for general software dependencies like Python, PyTorch, or CUDA. |
| Experiment Setup | Yes | We conduct full-parameter fine-tuning on the augmented datasets, initializing from the Llama-3.1-8B-Instruct checkpoint. ... The model is fine-tuned for 1 epoch with a maximum sequence length of 3k tokens. ... Table 5: Hyperparameters for training Co RAG. Multi-hop QA: Learning rate 5e-6, Batch size 256. KILT Benchmark: Learning rate 1e-5, Batch size 1024. Warmup steps 100, # Retrieved passages 20. ... To balance the three loss terms in Section 3.2, we set a sample ratio of 0.2 for both the sub-query and sub-answer generation tasks. |