Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
UMoE: Unifying Attention and FFN with Shared Experts
Authors: Yuanhang Yang, Chaozheng Wang, Jing Li
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To evaluate the effectiveness of UMo E, we conduct extensive experiments across various model sizes and tasks, including pre-training and zero-shot evaluations. With the reformulated attention mechanism, the attention-based Mo E layers of UMo E match or exceed the performance of previous FFN-based Mo E layers. ... From Table 1, we observe that UMo E shows consistent superiority across different model sizes and datasets. In the base model regime, UMo E achieves the best performance. |
| Researcher Affiliation | Academia | 1Institute of Science Tokyo, Tokyo, Japan 2The Chinese University of Hong Kong, Hong Kong, China 3Hong Kong Polytechnic University, Hong Kong, China |
| Pseudocode | Yes | Figure 3: Implementation details of a UMo E layer. The input consists of a sequence X containing n token hidden states and x representing the final hidden state. For simplicity, this implementation focuses on computing the output for the last token. |
| Open Source Code | Yes | Our code is available at https://github.com/ysngki/UMo E. |
| Open Datasets | Yes | We conduct language modeling pretraining on two datasets: Fine Web-Edu 100B [29] and Wikitext-103 [30]. ... The datasets used by this paper are all publicly available. |
| Dataset Splits | No | Datasets. We conduct language modeling pretraining on two datasets: Fine Web-Edu 100B [29] and Wikitext-103 [30]. ... Unless otherwise specified, models are pretrained on 50B tokens from Fine Web-Edu with batch size of 1024. For Wikitext-103, following Csordás et al. [13], models are trained for 100k steps... The zero-shot performance of models trained on Fine Web-Edu is evaluated using the lm-evaluation-harness framework [32]. |
| Hardware Specification | Yes | All experiments were conducted on a single NVIDIA H100 GPU. |
| Software Dependencies | No | We apply the LLa MA tokenizer [31] with a 32K vocabulary size to both datasets. ... The load balancing loss proposed by Switch Transformer [5] is adopted to encourage a balanced load across experts. ... MACs is measured using the Deep Speed Flops Profiler. |
| Experiment Setup | Yes | We evaluate two model configurations: The base models comprise 12 layers with a hidden size of 768, while the large models consist of 24 layers with a hidden size of 2048. ... models are pretrained on 50B tokens from Fine Web-Edu with batch size of 1024. For Wikitext-103, following Csordás et al. [13], models are trained for 100k steps... Detailed hyperparameters are provided in A.4. |