Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Linear Attention for Efficient Bidirectional Sequence Modeling
Authors: Arshia Afzal, Elias Abad Rocamora, Leyla Candogan, Pol Puigdemont, Francesco Tonin, Yongtao Wu, Mahsa Shoaran, Volkan Cevher
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To validate the speed and efficiency of LION we focused on well-known bidirectional sequence modeling tasks: Image Classification on Image Net-1K [38] and Masked Language Modeling (MLM) on the C4 dataset [8]. We also conduct experiments on the LRA dataset to ensure the stability of LION. |
| Researcher Affiliation | Academia | Arshia Afzal, Elias Abad Rocamora, Leyla Naz Candogan, Pol Puigdemont, Francesco Tonin, Yongtao Wu, and Volkan Cevher are with LIONS@EPFL. Mahsa Shoaran is with Integrated Neurotechnologies Laboratory (INL)@EPFL. |
| Pseudocode | Yes | Below we present the Python code used for the creation of the bidirectional mask M as described in previous sections. #caption=Code for generation of the selective bidirectional mask of \lion , 2 def create_matrix_from_tensor (tensor): 3 cumsum = torch.exp(torch.cumsum(tensor , dim=-1)) 4 A = torch.matmul(cump.unsqueeze(-1) , 5 1/ ( cump.unsqueeze(-1).transpose(-1,-2))) |
| Open Source Code | Yes | Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Code is available. |
| Open Datasets | Yes | To validate the speed and efficiency of LION we focused on well-known bidirectional sequence modeling tasks: Image Classification on Image Net-1K [38] and Masked Language Modeling (MLM) on the C4 dataset [8]. We also conduct experiments on the LRA dataset to ensure the stability of LION. |
| Dataset Splits | Yes | We employ the C4 dataset [8], a maximum sequence length during pretraining of 128 and a masking probability of 0.3 and 0.15 for the training and validation sets respectively. We train our model for 70, 000 steps with a batch size of 4096. |
| Hardware Specification | Yes | All experiments were conducted on a single machine for CIFAR-100 and multiple machines for Image Net, using NVIDIA A100 SXM4 80GB GPUs. |
| Software Dependencies | No | The paper mentions "PyTorch" implicitly through "PyTorch matrix operations" (page 2), "M2 repository" and "nano GPT4" but does not specify exact version numbers for any software libraries or frameworks. Therefore, the information is insufficient to confirm specific software dependencies with versions. |
| Experiment Setup | Yes | All our pretraining hyperparameters follow Fu et al. [10]: We employ the C4 dataset [8], a maximum sequence length during pretraining of 128 and a masking probability of 0.3 and 0.15 for the training and validation sets respectively. We train our model for 70, 000 steps with a batch size of 4096. We employ the decoupled Adam W optimizer with a learning rate of 8 × 10−4, β1 = 0.9, β2 = 0.98, ϵ = 10−6 and weight decay 10−5. As a scheduler, we perform a linear warm-up for 6% of the training steps and a linear decay for the rest of training until reaching 20% of the maximum learning rate. |