The Value-Improvement Path: Towards Better Representations for Reinforcement Learning
Authors: Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Marc G. Bellemare, David Silver7160-7168
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To test our hypothesis empirically, we augmented a standard deep RL agent with an auxiliary task of learning the value-improvement path. In a study of Atari 2600 games, the augmented agent achieved approximately double the mean and median performance of the baseline agent. Our goal in this section is to empirically study the effect of the previously discussed auxiliary tasks on the quality of the learned representation. For these experiments, we use the Atari-57 benchmark from the Arcade Learning Environment (Bellemare et al. 2013, ALE). |
| Researcher Affiliation | Industry | 1 Deep Mind 2 Google Research wdabney@google.com |
| Pseudocode | No | The paper does not contain any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not contain any explicit statement about releasing source code for the described methodology, nor does it provide a link to a code repository. |
| Open Datasets | Yes | For these experiments, we use the Atari-57 benchmark from the Arcade Learning Environment (Bellemare et al. 2013, ALE). |
| Dataset Splits | No | The paper mentions evaluating on a 'held-out set of transitions' but does not specify exact percentages or counts for train/validation/test splits, nor does it cite a predefined standard split for reproduction. |
| Hardware Specification | No | The paper does not specify any hardware details such as GPU models, CPU types, or cloud computing resources used for running the experiments. |
| Software Dependencies | No | The paper mentions algorithms like 'Double DQN' and 'Rainbow agent' but does not provide specific software library names with version numbers (e.g., PyTorch 1.9, TensorFlow 2.x) or other ancillary software dependencies with versions. |
| Experiment Setup | Yes | While training each agent, for 200 million environment frames, we saved the current network every 2 million frames. Each auxiliary task is trained as a linear function of the last hidden layer of the neural network used by Double DQN. We generated the cumulants for Cumulant Values and Cumulant Policies using a random network (details in Appendix C). |