Conditions on Features for Temporal Difference-Like Methods to Converge
Authors: Marcus Hutter, Samuel Yang-Zhao, Sultan Javed Majeed
IJCAI 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | Our main contribution is to prove that natural algorithms, even under the setting where the value function can be represented exactly by the features, are inherently prone to nonuniqueness and will converge to the wrong solution for most feature choices. Our main result is as follows: Theorem 5.1. Natural algorithms converge if and only if all non-zero linear combination of the features achieve their extreme values on a sub-region of the state space that has nonzero measure under the stationary distribution. |
| Researcher Affiliation | Academia | Marcus Hutter1 , Samuel Yang-Zhao1 and Sultan Javed Majeed1 1Australian National University {marcus.hutter, u6642247, sultan.majeed}@anu.edu.au |
| Pseudocode | No | The paper contains mathematical formulations, definitions, and theorems but no pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | No | The paper does not provide an explicit statement about the release of source code or links to a code repository for the described methodology. |
| Open Datasets | No | The paper is theoretical and does not use or reference any specific datasets for training experiments. |
| Dataset Splits | No | The paper is theoretical and does not report on experimental validation, thus no dataset splits for validation are provided. |
| Hardware Specification | No | The paper does not provide any specific details about hardware used for experiments. |
| Software Dependencies | No | The paper does not provide specific software dependencies or version numbers. |
| Experiment Setup | No | The paper is theoretical and does not describe any experimental setup details such as hyperparameters or training configurations. |