Almost Horizon-Free Structure-Aware Best Policy Identification with a Generative Model
Authors: Andrea Zanette, Mykel J. Kochenderfer, Emma Brunskill
NeurIPS 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | This paper focuses on the problem of computing an ǫ-optimal policy in a discounted Markov Decision Process (MDP) provided that we can access the reward and transition function through a generative model. We propose an algorithm that is initially agnostic to the MDP but that can leverage the specific MDP structure, expressed in terms of variances of the rewards and next-state value function, and gaps in the optimal action-value function to reduce the sample complexity needed to find a good policy, precisely highlighting the contribution of each state-action pair to the final sample complexity. A key feature of our analysis is that it removes all horizon dependencies in the sample complexity of suboptimal actions except for the intrinsic scaling of the value function and a constant additive term. |
| Researcher Affiliation | Academia | Andrea Zanette Institute for Computational and Mathematical Engineering, Stanford University, CA zanette@stanford.edu Mykel J. Kochenderfer Department of Aeronautics and Astronautics, Stanford University, CA mykel@stanford.edu Emma Brunskill Department of Computer Science, Stanford University, CA ebrun@cs.stanford.edu |
| Pseudocode | Yes | Algorithm 1 BESPOKE |
| Open Source Code | No | The paper does not provide any statements or links regarding the availability of open-source code for the described methodology. |
| Open Datasets | No | This paper is theoretical and does not use or reference any datasets for training or evaluation. |
| Dataset Splits | No | This paper is theoretical and does not conduct experiments with dataset splits. Therefore, no validation split information is provided. |
| Hardware Specification | No | This paper is theoretical and focuses on algorithm design and sample complexity analysis. It does not report on experimental setups or hardware used. |
| Software Dependencies | No | This paper is theoretical and focuses on algorithm design and analysis. It does not mention any specific software dependencies with version numbers required for implementation or experiments. |
| Experiment Setup | No | This paper is theoretical and does not report on experiments with specific setups or hyperparameters. |