Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Depth-Width Tradeoffs for Transformers on Graph Tasks
Authors: Gilad Yehudai, Clayton Sanford, Maya Bechler-Speicher, Orr Fischer, Ran Gilad-Bachrach, Amir Globerson
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically investigate these trade-offs between the relative powers of depth and width and find tasks where wider models have the same accuracy as deep models, while having much faster train and inference time due to parallelizable hardware. 6 Experiments We showcase the effect of width growth in transformers by training a variety of models on synthetic graph algorithmic tasks. |
| Researcher Affiliation | Collaboration | Gilad Yehudai Courant Institute of Mathematical Sciences, New York University EMAIL, Clayton Sanford Google Research, Maya Bechler-Speicher Meta AI, Or Fischer Bar-Ilan University, Ran Gilad-Bachrach Department of Bio-Medical Engineering, Edmond J. Safra Center for Bioinformatics, Tel-Aviv University Tel Aviv University, Amir Globerson Google Research Tel-Aviv University |
| Pseudocode | No | The paper only describes algorithmic steps in prose within the proofs (e.g., 'The proof idea is to embed all the information about the graph into a single token, and then offload the main bulk of the solution to the MLP. For that, the first layer will transform the input of each node from adjacency rows to only indicate its two neighbors.'). No structured pseudocode or algorithm block is present. |
| Open Source Code | Yes | 5Code is provided in the Supplementary Material. |
| Open Datasets | Yes | We compared the adjacency-row representation to the edge-list representation by training a transformer model on three Open Graph Benchmark (OGB) Hu et al. [2020] datasets: ogbg-molhiv, ogbg-molbbbp, and ogbg-molbace. [...] Here we list multiple such datasets from the Open Graph Benchmark (OGB) Hu et al. [2020] as well as TUdatasets Morris et al. [2020]. |
| Dataset Splits | Yes | For the connectivity dataset, [...] Each type of graph is sampled in equal proportions, shuffled, and split into training, validation, and test sets to maintain class balance. [...] In Section 6.3 we used three molecular property prediction datasets from Open Graph Benchmark (OGB) Hu et al. [2020]. |
| Hardware Specification | No | The paper mentions 'the ability of GPUs to parallelize the computations' but does not provide specific hardware details such as GPU models, CPU models, or memory specifications used for the experiments. |
| Software Dependencies | No | In our experiments, we used a standard transformer architecture using Pytorch s transformer encoder layers [Paszke et al., 2019]. |
| Experiment Setup | Yes | We trained a transformer with a fixed amount of 100k parameters split between varying depth and width. We examine the following pairs of (depth, width): (1, 125), (2, 89), (4, 63), (8, 45), (10, 40). We train each model for 100 epochs... For all experiments, we use a fixed drouput rate of 0.1 and Relu activations. In Section 6.1 we tuned the learning rate in {10 4, 5 10 5}, batch size in {32, 64}. In Section 6.3 we tuned the learning rate in {10 3, 5 10 3}, number of layers in {3, 5, 6, 10, 12}, hidden dimensions in {32, 64}. We used batch size of size 64. |