The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization
Authors: Mufan Li, Mihai Nica, Dan Roy
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Using simulations, we show that the SDE closely matches the distribution of the random covariance matrix of finite networks. Additionally, we recover an if-and-only-if condition for exploding and vanishing norms of large shaped networks based on the activation function. We provide simulations to verify theoretical predictions and help interpret properties of real DNNs. See Figures 1 and 2 and supplemental simulations in Appendix F. |
| Researcher Affiliation | Academia | Mufan (Bill) Li University of Toronto, Vector Institute Mihai Nica University of Guelph, Vector Institute Daniel M. Roy University of Toronto, Vector Institute |
| Pseudocode | No | No pseudocode or algorithm blocks found. |
| Open Source Code | Yes | All of our simulations (at initialization only) are contained the file Correlation.ipynb. |
| Open Datasets | No | The paper describes simulations, not experiments on a publicly available dataset. No concrete access information for a dataset used for training or evaluation is provided. |
| Dataset Splits | No | No specific dataset splits (training, validation, test) are mentioned, as the paper focuses on simulations rather than using a standard dataset. |
| Hardware Specification | No | Our simulations were small enough that it did not require GPUs. No specific hardware details (e.g., CPU model, memory) are provided. |
| Software Dependencies | No | No specific software dependencies with version numbers are mentioned. |
| Experiment Setup | Yes | Simulation details: n = d = 150, ρ0 = 0.3, 213 samples for each. In right column: c+ = 0, c = 1, DE step size 1e 2. |