Deterministic Policy Gradient Algorithms
Authors: David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller
ICML 2014 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We apply our deterministic actor-critic algorithms to several benchmark problems: a high-dimensional bandit; several standard benchmark reinforcement learning tasks with low dimensional action spaces; and a high-dimensional task for controlling an octopus arm. Our results demonstrate a significant performance advantage to using deterministic policy gradients over stochastic policy gradients, particularly in high dimensional tasks. |
| Researcher Affiliation | Collaboration | David Silver DAVID@DEEPMIND.COM Deep Mind Technologies, London, UK Guy Lever GUY.LEVER@UCL.AC.UK University College London, UK Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller *@DEEPMIND.COM Deep Mind Technologies, London, UK |
| Pseudocode | No | The paper describes algorithms using equations and text, but does not include structured pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | No | A video of an 8 segment arm, trained by COPDAC-Q, is also available.4 http://www0.cs.ucl.ac.uk/staff/D.Silver/ web/Applications.html. This link points to a video/applications page, not directly to the source code for the methodology described. |
| Open Datasets | No | The paper mentions benchmark problems like 'continuous bandit', 'mountain car', 'pendulum', '2D puddle world', and 'octopus arm' tasks, which are simulated environments. It does not provide concrete access information (specific link, DOI, repository name, formal citation with authors/year) for a publicly available or open dataset. |
| Dataset Splits | No | The paper mentions performing 'parameter sweep over all step-size parameters and variance parameters', which indicates hyperparameter tuning, but it does not provide specific dataset split information (exact percentages, sample counts, citations to predefined splits, or detailed splitting methodology) needed to reproduce data partitioning. |
| Hardware Specification | No | The paper does not provide specific hardware details (exact GPU/CPU models, processor types with speeds, memory amounts, or detailed computer specifications) used for running its experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details (e.g., library or solver names with version numbers like Python 3.8, CPLEX 12.4) needed to replicate the experiment. |
| Experiment Setup | Yes | The discount was γ = 0.99 for mountain car and pendulum and γ = 0.999 for puddle world. Actions outside the legal range were capped. We performed a parameter sweep over step-size parameters; variance was initialised to 1/2 the legal range. ... using a sigmoidal multi-layer perceptron (8 hidden units and sigmoidal output units) to represent the policy µ(s). The advantage function Aw(s, a) was represented by compatible function approximation (see Section 4.3), while the state value function V v(s) was represented by a second multi-layer perceptron (40 hidden units and linear output units). |