Pruning neural networks without any data by iteratively conserving synaptic flow
Authors: Hidenori Tanaka, Daniel Kunin, Daniel L. Yamins, Surya Ganguli
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We first mathematically formulate and experimentally verify a conservation law that explains why existing gradient-based pruning algorithms at initialization suffer from layer-collapse, the premature pruning of an entire layer rendering a network untrainable. ... and demonstrate empirically2 that this algorithm achieves state-of-the-art pruning performance on 12 distinct combinations of models and datasets (Sec. 7). |
| Researcher Affiliation | Collaboration | Hidenori Tanaka Physics & Informatics Laboratories NTT Research, Inc. Department of Applied Physics Stanford University Daniel Kunin Institute for Computational and Mathematical Engineering Stanford University Daniel L. K. Yamins Department of Psychology Department of Computer Science Stanford University Surya Ganguli Department of Applied Physics Stanford University |
| Pseudocode | Yes | Algorithm 1: Iterative Synaptic Flow Pruning (Syn Flow). Input: network f(x; 0), compression ratio , iteration steps n 0: f(x; 0) ; .Set model to eval modea 1: µ = 1 ; .Initialize binary mask for k in [1, . . . , n] do 2: µ µ 0 ; .Mask parameters 1 ; .Evaluate Syn Flow objective @ µ µ ; .Compute Syn Flow score 5: (1 k/n) percentile of S ; .Find threshold 6: µ ( < S) ; .Update mask end 7: f(x; µ 0) ; .Return masked network |
| Open Source Code | Yes | All code is available at github.com/ganguli-lab/Synaptic-Flow. |
| Open Datasets | Yes | datasets (CIFAR-10/100 and Tiny Image Net) |
| Dataset Splits | No | No explicit dataset split percentages, sample counts, or detailed splitting methodology for training, validation, and testing were provided in the main text. |
| Hardware Specification | No | No specific hardware details such as GPU/CPU models or detailed compute environment specifications were provided. |
| Software Dependencies | No | No specific software dependencies with version numbers (e.g., library names with versions) were mentioned in the paper. |
| Experiment Setup | No | The paper states 'See Appendix 13 for more details and hyperparameters of the experiments' but these details are not present in the provided main text. |