Rank Diminishing in Deep Neural Networks
Authors: Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, Zheng-Jun Zha
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | By virtue of our numerical tools, we provide the first empirical analysis of the per-layer behavior of network rank in practical settings, i.e., Res Nets, deep MLPs, and Transformers on Image Net. These empirical results are in direct accord with our theory. |
| Researcher Affiliation | Collaboration | Ruili Feng1, Kecheng Zheng2,1, Yukun Huang1, Deli Zhao2,3, Michael Jordan4, Zheng-Jun Zha1 1University of Science and Technology of China, Hefei, China 2Ant Research, 3Alibaba Group, Hangzhou, China 4University of California, Berkeley |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code to detect the rank behavior of networks can be found in https://github.com/RuiLiFeng/Rank-Diminishing-in-Deep-Neural-Networks. |
| Open Datasets | Yes | In this section, we numerically validate our theory in three types of architectures of benchmark deep neural networks, CNNs, MLPs, and Transformers, in the Image Net [14] data domain. |
| Dataset Splits | Yes | The classification accuracy on the validation set of using Eq. (12), instead of the true logits, to predict the label is reported in blue (if tested on positive samples only, the accuracy rates are 98%, 90%, 82% for cases in (b,c,d) correspondingly). |
| Hardware Specification | No | Computing the full Jacobian representation of sub-networks of Res Net-50, for example, consumes over 150G GPU memory and several days at a single input point. This statement mentions 'GPU memory' but does not specify a particular GPU model or other hardware components. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers. |
| Experiment Setup | No | Details of the experiment setup can be found in Appendix E. However, Appendix E is not provided in the given text, so the specific details are not present in the main body. |