Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Wavy Transformer
Authors: Satoshi Noguchi, Yoshinobu Kawahara
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Furthermore, to demonstrate the effectiveness and generality of Wavy Transformer, we conduct extensive experiments on NLP, CV, and sparse-graph tasks. |
| Researcher Affiliation | Academia | Satoshi Noguchi Research Institute for Value-Added Information Generation, JAMSTEC Center for Advanced Intelligence Project, RIKEN EMAIL Yoshinobu Kawahara Graduate School of Information Science and Technology, The University of Osaka Center for Advanced Intelligence Project, RIKEN EMAIL |
| Pseudocode | No | The paper describes mathematical equations for the Wavy Transformer block (e.g., Eqs. 9, 10, 11) and provides a schematic diagram in Figure 1, but it does not contain explicit pseudocode or algorithm blocks with structured steps. |
| Open Source Code | Yes | Source code and models are available at https://github.com/noguchisatoshi/Wavy-Transformer. |
| Open Datasets | Yes | The GLUE [42] benchmark provides a comprehensive suite of natural language understanding tasks that serve as critical tests for evaluating the performance of pretrained language models. All of our experiments are conducted on the Image Net dataset [10] including around 1k classes 1.3M images in the training dataset and 50k images in the validation set. We further evaluate on OGBN-Arxiv and OGBN-Proteins [23] using DIFFormer [44] under the authors training protocol and hyperparameters. |
| Dataset Splits | Yes | Following [11], we construct a English corpus from Wikipedia [11] and Books Corpus [50], splitting it into 95 % training and 5 % validation data. All of our experiments are conducted on the Image Net dataset [10] including around 1k classes 1.3M images in the training dataset and 50k images in the validation set. |
| Hardware Specification | Yes | Experiments used servers with eight V100 GPUs, four L40 GPUs, or four H100 GPUs, selected according to task size. |
| Software Dependencies | No | The paper mentions software components like 'Adam W [27]' as an optimizer (Appendix A.1.1) but does not provide specific version numbers for these or other key software dependencies like Python, PyTorch, or CUDA. |
| Experiment Setup | Yes | We implement three types of models with different residual connections such as diffuse, wave, and mix in a 24-layer BERT with hidden size 256 and 4 attention heads based on Post-LN Wavy Transformer blocks. As a optimizer, we use Adam W [27] with learning rate 5.0 10 5, warm-up for first 10 % of steps, then linear decay to step 10k. The hyper-parameters of various downstream tasks are shown in Table 8. Table 8: Hyper-parameters for different downstream tasks. Batch size 32 16 Weight decay [0.1, 0.01] [0.1, 0.01] Warmup proportion 0.1 0.1 Learning rate decay linear linear Training epochs 3 3 Learning rate 0.00005 0.00005 |