Recipe for a General, Powerful, Scalable Graph Transformer
Authors: Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, Dominique Beaini
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We test our architecture on 16 benchmarks and show highly competitive results in all of them, show-casing the empirical benefits gained by the modularity and the combination of different strategies. We perform ablation studies on 4 datasets to evaluate the contribution of the message-passing module, the global attention module, and the positional or structural encodings. Then, we evaluate GPS on a diverse set of 11 benchmarking datasets, and show state-of-the-art (SOTA) results in many cases. |
| Researcher Affiliation | Collaboration | Ladislav Rampášek Mila, Université de Montréal Mikhail Galkin Mila, Mc Gill University Vijay Prakash Dwivedi Nanyang Technological University, Singapore Anh Tuan Luu Nanyang Technological University, Singapore Guy Wolf Mila, Université de Montréal Dominique Beaini Valence Discovery, Mila, Université de Montréal |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | The source code of GRAPHGPS is available at: https://github.com/rampasek/Graph GPS. |
| Open Datasets | Yes | We test on datasets from different sources to ensure diversity, providing their detailed description in Appendix A.1. From the Benchmarking GNNs [15], we test on the ZINC, PATTERN, CLUSTER, MNIST, CIFAR10. From the open graph benchmark (OGB) [27], we test on all graph-level datasets: ogbg-molhiv, ogbg-molpcba, ogbg-code2, and ogbg-ppa, and from their large-scale challenge we test on the OGB-LSC PCQM4Mv2 [28]. Finally, we also select Mal Net-Tiny [21] with 5000 graphs, each of up to 5000 nodes, since the number of nodes provide a scaling challenge for Transformers. |
| Dataset Splits | Yes | We describe the datasets in Appendix A.1, splits in Appendix A.2, hyperparameters in Appendix A.3. Full configuration files are provided in the supplementary material. |
| Hardware Specification | Yes | SAN on COCO-SP exceeded 60h time limit on an NVidia A100 GPU system. We elaborate on the compute and used resources in Appendix A.4. |
| Software Dependencies | No | The paper mentions software like Py G and Graph Gym, but does not provide specific version numbers for them in the main text. |
| Experiment Setup | Yes | We abide to the ~500k model parameter budget and closely follow the experimental setup and hyperparameter choices of the graph Transformer baselines tested in LRGB [17]. We keep the same node/edge encoders and model depth (number of layers), deviating only in two aspects: i) we slightly decrease the size of hidden node representations to fit within the parameter budget, ii) we employ cosine learning rate schedule as in our other experiments (Section A.3). We describe the datasets in Appendix A.1, splits in Appendix A.2, hyperparameters in Appendix A.3. Full configuration files are provided in the supplementary material. |