Sobolev Training for Neural Networks
Authors: Wojciech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, Razvan Pascanu
NeurIPS 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide theoretical justifications for such an approach as well as examples of empirical evidence on three distinct domains: regression on classical optimisation datasets, distilling policies of an agent playing Atari, and on large-scale applications of synthetic gradients. |
| Researcher Affiliation | Industry | Wojciech Marian Czarnecki, Simon Osindero, Max Jaderberg Grzegorz Swirszcz, and Razvan Pascanu Deep Mind, London, UK {lejlot,osindero,jaderberg,swirszcz,razp}@google.com |
| Pseudocode | No | The paper includes diagrams of compute graphs (Figure 1), but no structured pseudocode or algorithm blocks are provided. |
| Open Source Code | No | The paper references Sonnet [1], a neural network library, and TensorFlow [2], indicating they were used for experiments. However, there is no explicit statement or link provided for the open-source code of the specific methodology (Sobolev Training) described in the paper. |
| Open Datasets | Yes | We consider three domains where information about derivatives is available during training... regression on classical optimisation datasets... distilling policies of an agent playing Atari [17]... training deep, complex models using synthetic gradients... CIFAR-10... Image Net [3]. |
| Dataset Splits | No | The paper mentions 'a hold-out test set' and train/test splits for specific experiments (e.g., 'percentage of the 100K recorded states used for training (the remaining are used for testing)'). However, it does not explicitly describe a separate validation dataset split with specific percentages or counts. |
| Hardware Specification | No | The paper states 'All experiments were performed using Tensor Flow [2] and the Sonnet neural network library [1]', but it does not specify any details about the hardware (e.g., GPU models, CPU types, memory) used for running the experiments. |
| Software Dependencies | No | The paper mentions 'Tensor Flow [2] and the Sonnet neural network library [1]' were used. However, it does not provide specific version numbers for these software components. |
| Experiment Setup | Yes | We train two hidden layer neural networks with 256 hidden units per layer with Re LU activations... We choose all the losses of Sobolev Training to be L2 errors... stochastic approximation... min θ DKL(π(s|θ) π (s)) + αEv [...]. The experiments presented use learning rates, annealing schedule, etc. optimised to maximize the backpropagation baseline (details in the SM). |