Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
REINFORCE Converges to Optimal Policies with Any Learning Rate
Authors: Samuel Robertson, Thang Chu, Bo Dai, Dale Schuurmans, Csaba Szepesvari, Jincheng Mei
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct several experiments to illustrate the convergence behavior of REINFORCE algorithm in the finite-horizon setting. Experiments are performed using a chain MDP (Fig. 1b)... Performance is evaluated by measuring the average suboptimality gap... Our first experiment (Fig. 3a) demonstrates the benefits of using a large learning rate. |
| Researcher Affiliation | Collaboration | 1University of Alberta 2Google Deep Mind 3Georgia Institute of Technology EMAIL EMAIL |
| Pseudocode | Yes | Algorithm 1 Stochastic gradient bandit algorithm and Algorithm 2 REINFORCE are presented with numbered steps. |
| Open Source Code | No | Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: Our experiments are simple enough that they can be recreated easily using the information provided in Section 5.5. We are studying classic algorithms and use only simulated data, which should be easily reproducible from the detailed descriptions. |
| Open Datasets | Yes | Experiments are performed using a chain MDP (Fig. 1b)... Next, we gradually increase the complexity of our evaluation by testing the REINFORCE algorithm (Algorithm 2) on the deep sea treasure environment... Finally, we evaluate the performance of the REINFORCE algorithm (Algorithm 2) in the Cartpole environment. |
| Dataset Splits | No | Experiments are performed using a chain MDP (Fig. 1b)... the REINFORCE algorithm is run for 105 episodes across 30 seeds. Performance is evaluated by measuring the average suboptimality gap... over the 30 seeds. |
| Hardware Specification | No | Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [No] Justification: The environments (Chain MDP and Tree MDP) are simple and the experiments are small enough to run on any modern machine (e.g. an Intel Macbook 2019). |
| Software Dependencies | No | Question: Does the paper provide SPECIFIC ANCILLARY SOFTWARE DETAILS (e.g., library or solver names with version numbers like Python 3.8, CPLEX 12.4) needed to replicate the experiment? Answer: [No] |
| Experiment Setup | Yes | The policy is parameterized using a softmax function, and parameters are initialized to 0 R|S| |A|. For each learning rate η, the REINFORCE algorithm is run for 105 episodes across 30 seeds... we evaluated REINFORCE algorithm with larger learning rates η {0.00001, 0.001, 0.1}. We further explore the effect of even larger learning rates η {0.5, 1, 2}... |