Finite-Time Analysis of Single-Timescale Actor-Critic
Authors: Xuyang Chen, Lin Zhao
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | We investigate the more practical online single-timescale actor-critic algorithm on continuous state space, where the critic assumes linear function approximation and updates with a single Markovian sample per actor step. Previous analysis has been unable to establish the convergence for such a challenging scenario. We demonstrate that the online single-timescale actor-critic method provably finds an ϵ-approximate stationary point with e O(ϵ 2) sample complexity under standard assumptions, which can be further improved to O(ϵ 2) under the i.i.d. sampling. Our novel framework systematically evaluates and controls the error propagation between the actor and critic. It offers a promising approach for analyzing other single-timescale reinforcement learning algorithms as well. |
| Researcher Affiliation | Academia | Xuyang Chen National University of Singapore chenxuyang@u.nus.edu Lin Zhao National University of Singapore elezhli@nus.edu.sg |
| Pseudocode | Yes | Algorithm 1 Single-timescale Actor-Critic |
| Open Source Code | No | The paper does not contain any statements or links indicating the availability of open-source code for the described methodology. |
| Open Datasets | No | The paper is theoretical and focuses on finite-time analysis and convergence proofs, not on empirical training using specific datasets. It mentions 'Markov decision process' and 'Markovian sampling' but does not specify a publicly available dataset for training. |
| Dataset Splits | No | The paper is theoretical and does not conduct experiments involving dataset splits for training, validation, or testing. |
| Hardware Specification | No | The paper is theoretical and does not describe any specific hardware used for experiments. |
| Software Dependencies | No | The paper is theoretical and does not specify any software dependencies with version numbers. |
| Experiment Setup | No | The paper is theoretical and focuses on mathematical analysis rather than detailing an experimental setup with hyperparameters or training settings. |