Decoupled Neural Interfaces using Synthetic Gradients
Authors: Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, Koray Kavukcuoglu
ICML 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section we perform empirical expositions of the use of DNIs and synthetic gradients, first by applying them to RNNs in Sect. 3.1 showing that synthetic gradients extend the temporal correlations an RNN can learn. Secondly, in Sect. 3.2 we show how a hierarchical, two-timescale system of networks can be jointly trained using synthetic gradients to propagate error signals between networks. Finally, we demonstrate the ability of DNIs to allow asynchronous updating of layers a feed-forward network in Sect. 3.3. |
| Researcher Affiliation | Industry | 1Deep Mind, London, UK. Correspondence to: Max Jaderberg <jaderberg@google.com>. |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide concrete access to source code for the methodology described. |
| Open Datasets | Yes | Copy and Repeat Copy We first look at two synthetic tasks Copy and Repeat Copy tasks from (Graves et al., 2014). [...] Language Modelling We also applied our DNI-enabled RNNs to the task of character-level language modelling, using the Penn Treebank dataset (Marcus et al., 1993). |
| Dataset Splits | Yes | We measure error in bits per character (BPC) as in (Graves, 2013), perform early stopping based on validation set error, and for simplicity do not perform any learning rate decay. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory amounts) used for running its experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details with version numbers. |
| Experiment Setup | Yes | The implementation details of the RNN models are given in Sect. D.2 in the Supplementary Material. [...] For full experimental details please refer to Sect. D.2 in the Supplementary Material. [...] Full experimental details can be found in Sect. D.3 in the Supplementary Material. [...] More details are given in the Supplementary Material. |