Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
On Reasoning Strength Planning in Large Reasoning Models
Authors: Leheng Sheng, An Zhang, Zijian Wu, Weixiang Zhao, Changshuo Shen, zhang yi, Xiang Wang, Tat-Seng Chua
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Here we conduct preliminary experiments using Deep Seek-R1-distilled-Qwen on the MATH [5] dataset, which contains math questions across five difficulty levels. As visualized in Figure 1, we summarize three key empirical findings from the activation distribution of the LRM: |
| Researcher Affiliation | Academia | 1National University of Singapore 2University of Science and Technology of China 3Harbin Institute of Technology |
| Pseudocode | No | The paper describes procedures and methods but does not contain explicit pseudocode or algorithm blocks formatted like code. |
| Open Source Code | Yes | Our code is available at https://github.com/Alpha Lab-USTC/LRM-plans-Co T. |
| Open Datasets | Yes | We conduct the linear regression experiments on the MATH [5] dataset, where math questions are divided into five groups according to their difficulty. We test the predictions differences on data pairs of non-overthink questions and overthink questions. These overthink questions are constructed by forcing LRMs to overthink on vanilla questions by adopting overthink attacks [57], where these vanilla questions are sampled from the Alpaca Eval dataset. We conduct activation steering experiments on two kinds of questions that LRMs tend to overthink: general language understanding dataset MMLU [60] and Level 1 questions on MATH500 [5]. |
| Dataset Splits | Yes | We randomly split the dataset with a ratio of 9:1 for training and testing. |
| Hardware Specification | Yes | We implement all the experiments on 8 NVIDIA A100 GPUs. The whole computational resource cost of this research is about 80 A100 GPU days, which is mainly spent on the answer generation. |
| Software Dependencies | No | To implement this Lasso regression, we use the Python package of scikit-learn [53]. For the answer generation, we use the v LLM [63] framework for acceleration. The specific version numbers for these software components are not provided. |
| Experiment Setup | Yes | For the answer generation, we use the v LLM [63] framework for acceleration. Following the suggestions of Deep Seek 3, we set the temperature as 0.6 to prevent endless repetitions, set the maximum new generation length as 16,384, and set the rollout number as 8. For reducing the overfitting, we set the regularization term α in Equation 1 as 10. |