Exponential expressivity in deep neural networks through transient chaos
Authors: Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, Surya Ganguli
NeurIPS 2016 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This theoretical prediction of equations (3) and (5) is quantitatively confirmed in simulations in Fig. 3. ... These highly nontrivial predictions of the metric and curvature evolution equations in (8) are quantitatively confirmed in simulations in Figure 4C-E. |
| Researcher Affiliation | Collaboration | 1Stanford University, 2Google Brain |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code to reproduce all results available at: https://github.com/ganguli-lab/deepchaos |
| Open Datasets | No | The paper uses simulations of random deep neural networks with weights and biases drawn from a Gaussian distribution, and propagates a simple manifold (a circle) through them. It does not mention using a named public dataset for training or evaluation. |
| Dataset Splits | No | The paper focuses on theoretical predictions validated by simulations of randomly initialized networks, rather than using traditional datasets with explicit training/validation/test splits. |
| Hardware Specification | No | The paper does not provide any specific hardware details such as GPU or CPU models used for running its simulations. |
| Software Dependencies | No | The paper does not specify any software dependencies with version numbers. |
| Experiment Setup | Yes | Consider a deep feedforward network with D layers... with Nl neurons in each layer l... weights Wl ij are drawn i.i.d. from a zero mean Gaussian with variance σ2 w/Nl 1, while the biases are drawn i.i.d. from a zero mean Gaussian with variance σ2 b... for the special case of a sigmoidal nonlinearity, φ(h) = tanh(h)... a sigmoidal network with 1000 hidden units. |