Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning
Authors: Honglin Lin, Qizhi Pei, Zhuoshi Pan, Yu Li, Xin Gao, Juntao Li, Conghui He, Lijun Wu
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments on our created Caco-1.3M dataset demonstrate that Caco-trained models achieve strong competitive performance on mathematical reasoning benchmarks, outperforming existing strong baselines. We evaluate Caco through extensive experiments on standard mathematical reasoning benchmarks. Models fine-tuned using our Caco-1.3M dataset achieve strong competitive performance. |
| Researcher Affiliation | Academia | 1Shanghai Jiao Tong University 2Shanghai Artificial Intelligence Laboratory 3Tsinghua University 4Renmin University of China 5Soochow University EMAIL |
| Pseudocode | No | The paper describes its methodology in text and through figures (e.g., Figure 2: An overview framework of Caco data generation), but does not include any explicitly labeled 'Pseudocode' or 'Algorithm' blocks. |
| Open Source Code | Yes | https://github.com/LHL3341/Caco |
| Open Datasets | Yes | https://huggingface.co/datasets/LHL3341/Caco-1.3M. Mathematical Problems. We collected a broad set of mathematical problems from multiple sources to ensure diversity, such as the MATH dataset [14] (7.5K), Deep Scale R [26] (40K), and Big Math [2] (251K). Algorithmic Problems. We sample 40K problems from the Kodcode [43] dataset. Evaluation Setup. Following the evaluation protocol of Dart Math [38], we evaluate on multiple popular benchmarks to show the advantages, including MATH [14], GSM8K [6], College Math [34], Deep Mind-Mathematics [31], Olympiad Bench-Math [13], and Theorem QA [5]. |
| Dataset Splits | Yes | MATH [14]: A benchmark of 12,500 high school math competition problems, with 7,500 for training and 5,000 for testing. GSM8K [6]: This dataset contains 8,792 high-quality grade school math word problems, with 7,473 for training and 1,319 for testing. All models were evaluated using a unified framework 4 under the zero-shot setting. |
| Hardware Specification | Yes | We deployed Qwen2.5-72B-Instruct on 4 A100 GPUs to generate code from the raw datasets. All experiments were conducted on a single machine equipped with 8 NVIDIA A100 GPUs. |
| Software Dependencies | No | The paper mentions using the 'Llama Factory framework' and 'Adam W optimizer [22]' but does not provide specific version numbers for these or other software libraries (e.g., Python, PyTorch). |
| Experiment Setup | Yes | All models are fine-tuned for 3 epoch using a learning rate of 5 10 6, a batch size of 128, and a cosine decay schedule with a warm-up ratio of 0.03. The maximum sequence length (cutoff) was set to 4096, and the weight decay was 0.1. For problem and solution generation, we followed the Qwen3-8B best practice [37]. Specifically, we used: Temperature = 0.7, Top P = 0.8, Top K = 20, Min P = 0, and enable_thinking = False. |