Distinguished In Uniform: Self-Attention Vs. Virtual Nodes
Authors: Eran Rosenbluth, Jan Tönshoff, Martin Ritzert, Berke Kisin, Martin Grohe
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We complement our theoretical findings with an empirical study on both synthetic and real-world datasets and observe that a uniformly-expressive model can (sometimes) be learned, and a uniforminexpressive model does not generalize in terms of graph size. and 5 EXPERIMENTS First, we empirically verify the difference between GPS and MPGNN-VN architectures using synthetic datasets based on Theorems 4.3 and 4.7. ... Second, we perform experiments on real-world datasets and show that neither architecture strictly outperforms the other overall. |
| Researcher Affiliation | Academia | Eran Rosenbluth1 , Jan T onshoff1 , Martin Ritzert2 , Berke Kisin1 & Martin Grohe1 1RWTH Aachen University, 2Georg-August-Universit at G ottingen |
| Pseudocode | No | The paper defines various models and components using mathematical notation (e.g., Equation 1 for Message Passing, definitions for Transformer Layer and GPS) but does not provide any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | We perform two kinds of experiments1. First, we empirically verify the difference between GPS and MPGNN-VN architectures using synthetic datasets based on Theorems 4.3 and 4.7. ... 1https://github.com/toenshoff/VN-vs-GT |
| Open Datasets | Yes | For this, we evaluate MPGNN+VNs on real-world benchmark datasets from both the recent LRGB collection (Dwivedi et al., 2022) and OGB (Hu et al., 2020a). and The LRGB dataset collection (Dwivedi et al., 2022) has recently been introduced as a set of benchmarks... Additionally, we incorporate two graph-level datasets from the OGB project (Hu et al., 2020a). Standard splits are part of these benchmarks. |
| Dataset Splits | Yes | For this, we evaluate MPGNN+VNs on real-world benchmark datasets from both the recent LRGB collection (Dwivedi et al., 2022) and OGB (Hu et al., 2020a). and The LRGB dataset collection (Dwivedi et al., 2022) has recently been introduced as a set of benchmarks... Additionally, we incorporate two graph-level datasets from the OGB project (Hu et al., 2020a). Standard splits are part of these benchmarks. |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., GPU/CPU models, memory) used for running the experiments. |
| Software Dependencies | No | The paper mentions software components like 'SGD-based optimization' and various GNN architectures (GCN, GINE/GIN, Gated GCN) but does not provide specific version numbers for any software dependencies. |
| Experiment Setup | Yes | Table 2: Hyperparameters for Gated GCN+VN. and Table 3: Hyperparameters for GCN+VN. These tables list specific values for 'lr', 'dropout', '#layers', 'hidden dim', 'head depth', 'batch size', and '#epochs' for various datasets. |