Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Near Optimal Policy Optimization via REPS
Authors: Aldo Pacchiano, Jonathan N Lee, Peter Bartlett, Ofir Nachum
NeurIPS 2021 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | In this paper we aim to fill this gap by providing guarantees and convergence rates for the sub-optimality of a policy learned using first-order optimization methods applied to the REPS objective. We first consider the setting in which we are given access to exact gradients and demonstrate how near-optimality of the objective translates to near-optimality of the policy. We then consider the setting of stochastic gradients and introduce a technique that uses generative access to the underlying Markov decision process to compute parameter updates that maintain favorable convergence to the optimal regularized policy. |
| Researcher Affiliation | Collaboration | Aldo Pacchiano Microsoft Research EMAIL Jonathan Lee Stanford University EMAIL Peter L. Bartlett UC Berkeley EMAIL Google EMAIL |
| Pseudocode | Yes | Algorithm 1 Relative Entropy Policy Search [Sketch]. Input: Initial iterate v0, accuracy level > 0, gradient optimization algorithm O. ... Algorithm 2 Biased Gradient Estimator Input Number of samples t. ... Algorithm 3 Biased Stochastic Gradient Descent Input Desired accuracy , learning rates { t}1 t=1, and number-of-samples function n : N ! N . |
| Open Source Code | No | Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [N/A] |
| Open Datasets | No | The paper is theoretical and does not mention using any specific datasets for training or provide access information for any dataset. |
| Dataset Splits | No | The paper is theoretical and does not mention specific dataset splits for training, validation, or testing. |
| Hardware Specification | No | Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers. |
| Experiment Setup | No | The paper is theoretical and does not describe a concrete experimental setup with hyperparameters or system-level training settings. |