Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
LLM Layers Immediately Correct Each Other
Authors: Arjun Patrawala, Jiahai Feng, Erik Jones, Jacob Steinhardt
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To begin, we characterize TLCM through a series of observational experiments in Section 4. First, we show that TLCM is not present at initialization but emerges gradually during pretraining, suggesting that it is a persistent characteristic of LLM training dynamics. Second, we find that TLCM fires most frequently on tokens with high contextual dependency, including numbers, dates, and punctuation. We hypothesize that TLCM is important for handling tokens with high contextual dependency. Finally, we show that TLCM is the combined effort of both attention and MLP. |
| Researcher Affiliation | Academia | Arjun Patrawala Jiahai Feng Erik Jones Jacob Steinhardt University of California, Berkeley EMAIL |
| Pseudocode | No | The paper describes methods and mathematical notations but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | 1Code is available at https://github.com/arjunpat/transformer-correction |
| Open Datasets | Yes | We then average these matrices element-wise across approximately 100,000 tokens, across random documents in Wiki Text [34] using Hugging Face Transformers [49] |
| Dataset Splits | Yes | We then average these matrices element-wise across approximately 100,000 tokens, across random documents in Wiki Text [34] using Hugging Face Transformers [49]: M = 1 t Mt. These documents include a variety of languages and code. ... This corpus contains 79k tokens. |
| Hardware Specification | No | We did not include this information because the vast majority of our experiments are not computationally intensive. The most computationally intensive experiment required computing Jacobians of transformer layers, for which we used a single GPU; we detail this in the appendix. |
| Software Dependencies | No | We then average these matrices element-wise across approximately 100,000 tokens, across random documents in Wiki Text [34] using Hugging Face Transformers [49]: M = 1 t Mt. |
| Experiment Setup | Yes | To this end, we measure the cosine similarity between at and b t(Ξ±at) at Ξ± { 1, 0.9, . . . , 0.9, 1}. |