Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Small Singular Values Matter: A Random Matrix Analysis of Transformer Models
Authors: Max Staats, Matthias Thamm, Bernd Rosenow
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This work analyzes singular-value spectra of weight matrices in pretrained transformer models to understand how information is stored at both ends of the spectrum. Using Random Matrix Theory (RMT) as a zero information hypothesis, we associate agreement with RMT as evidence of randomness and deviations as evidence for learning. Surprisingly, we observe pronounced departures from RMT not only among the largest singular values the usual outliers but also among the smallest ones. A comparison of the associated singular vectors with the eigenvectors of the activation covariance matrices shows that there is considerable overlap wherever RMT is violated. Thus, significant directions in the data are captured by small singular values and their vectors as well as by the large ones. We confirm this empirically: zeroing out the singular values that deviate from RMT raises language-model perplexity far more than removing values from the bulk, and after fine-tuning the smallest decile can be the third most influential part of the spectrum. To explain how vectors linked to small singular values can carry more information than those linked to larger values, we propose a linear random-matrix model. Our findings highlight the overlooked importance of the low end of the spectrum and provide theoretical and practical guidance for SVD-based pruning and compression of large language models. |
| Researcher Affiliation | Academia | Max Staats Center for Scalable Data Analytics and Artificial Intelligence Leipzig University EMAIL Matthias Thamm Institute for Theoretical Physics Leipzig University EMAIL Bernd Rosenow Institute for Theoretical Physics Leipzig University EMAIL |
| Pseudocode | No | The paper includes mathematical equations for SVD, attention mechanisms, and the theoretical Marchenko-Pastur distribution, as well as descriptions of experimental procedures. However, it does not feature any explicitly labeled 'Pseudocode' or 'Algorithm' blocks, nor does it present structured steps in a code-like format. |
| Open Source Code | Yes | All code to generate the figures is open source and available under [2]. [2] Author Anonymous. All code, scripts, and data used in this work are included in a Zenodo archive: https://zenodo.org/records/15409885. Zenodo, 2025. doi: 10.5281/zenodo. 15409885. |
| Open Datasets | Yes | We study the singular value spectra of weight matrices from three pretrained models: Bert1 [26], Pythia2 [9], and Llama-8B3 [14]. We estimate C from the Wiki Text [33] dataset. The appendix shows additional results from the Book Corpus [53] datasets. These findings are further backed up by Llama results on the GSM8K benchmark... For Bert we additionally conduct fine-tuning experiments on Bool Q [13], rte [8, 19, 21] and SST2 [42]. We provide further experiments for removing deciles from all weight matrix types for LLa MA 3.1 8B Chat on the RULER [22] benchmark... |
| Dataset Splits | No | The paper mentions evaluating perplexity on the Wiki Text and Book Corpus datasets, and conducting fine-tuning experiments on Bool Q, RTE, and SST2. It also refers to results on GSM8K, RULER, and Human Eval benchmarks. While these are standard datasets often with predefined splits, the paper does not explicitly state the specific train/test/validation splits used for its experiments (e.g., 80/10/10 percentages, sample counts, or specific predefined split names from citations). For instance, it says 'we compute the perplexity on subparts of the Wiki Text and Book Corpus dataset' which is not specific enough. |
| Hardware Specification | Yes | Computing spectra and the activation covariance matrices for all models can be estimated as less than 100 GPU hours on a V100. |
| Software Dependencies | No | The paper does not explicitly list any software dependencies with specific version numbers, such as programming languages, libraries, or frameworks (e.g., Python 3.x, PyTorch 1.x, TensorFlow 2.x). |
| Experiment Setup | Yes | To compare the relative impact of different parts of the spectrum, we group the rank-ordered singular values of each matrix into ten equal-sized deciles, with the smallest 10% in the first decile and the largest 10% in the tenth. We apply this procedure by choosing one matrix type (e.g., Query) and zeroing out one of the ten deciles in all weight matrices of that type. We then reconstruct the weight matrix W using the original singular vectors. For the case of perfect learning, i.e. β , the outlier singular value approaches λ q 1 αq α λ2 , only differing from λ of W0 by the level repulsion with the bulk. The results in both panels are computed for α = 1, λ = 0.2, N = 2048, and K = 512. |