Reanalysis of Variance Reduced Temporal Difference Learning
Authors: Tengyu Xu, Zhe Wang, Yi Zhou, Yingbin Liang
ICLR 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we provide numerical results to verify our theoretical results. Note that in Appendix A, we provide further experiments on two problems in Open AI Gym Brockman et al. (2016) and one experiment to demonstrate that VRTD is more sample-efficient than vanilla TD. |
| Researcher Affiliation | Academia | Department of ECE, The Ohio State University ,Columbus, OH 43210, USA Department of ECE, The University of Utah, Salt Lake City, UT 84112, USA |
| Pseudocode | Yes | Algorithm 1 Variance Reduced TD with iid samples; Algorithm 2 Variance Reduced TD with Markovian samples Korda and La (2015) |
| Open Source Code | No | The paper does not provide any statements or links indicating that the source code for the described methodology is publicly available. |
| Open Datasets | Yes | In this section, we assess the practical performance of VRTD (Algorithm 2) on two problems in Open AI Gym Brockman et al. (2016), which are Frozen Lake (4x4) and Mountain Car. |
| Dataset Splits | No | The paper describes sampling data from MDP trajectories or stationary distributions and mentions independent runs, but does not specify explicit training, validation, or test dataset splits (e.g., percentages or counts) for reproducibility of data partitioning. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware (e.g., GPU/CPU models, memory) used to conduct the experiments. |
| Software Dependencies | No | The paper mentions using 'Open AI Gym' but does not specify any software dependencies with version numbers (e.g., Python, PyTorch, specific library versions) that would be needed to replicate the experiments. |
| Experiment Setup | Yes | In both experiments, we set the constant stepsize to be alpha = 0.1 and we run the experiments for five different batch sizes: M = 1, 50, 500, 1000, 2000. For Frozen Lake, 'We set the stepsize to be alpha = 0.1'. For Mountain Car, 'We set the stepsize to be alpha = 0.2 and run vanilla TD (M = 1) and VRTD with batch size M = 1000'. |