Graph External Attention Enhanced Transformer
Authors: Jianqing Liang, Min Chen, Jiye Liang
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments on benchmark datasets demonstrate that GEAET achieves stateof-the-art empirical performance. The source code is available for reproducibility at: https: //github.com/icm1018/GEAET. In this section, we evaluate the empirical performance of GEANet and GEAET on a variety of graph datasets with graph prediction and node prediction tasks, including CIFAR10, MNIST, PATTERN, CLUSTER and ZINC from Benchmarking GNNs (Dwivedi et al., 2020), as well as Pascal VOC-SP, COCO-SP, Petides-Struct, Petides-Func and PCQM-Contact from Long Range Graph Benchmark (LRGB; Dwivedi et al., 2022b), and the Tree Neighbour Match dataset (Alon & Yahav, 2021). |
| Researcher Affiliation | Academia | Jianqing Liang 1 Min Chen 1 Jiye Liang 1 1Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, School of Computer and Information Technology, Shanxi University, Taiyuan 030006, Shanxi, China. |
| Pseudocode | No | The paper describes the model architecture and components but does not provide structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | The source code is available for reproducibility at: https: //github.com/icm1018/GEAET. |
| Open Datasets | Yes | In this section, we evaluate the empirical performance of GEANet and GEAET on a variety of graph datasets with graph prediction and node prediction tasks, including CIFAR10, MNIST, PATTERN, CLUSTER and ZINC from Benchmarking GNNs (Dwivedi et al., 2020), as well as Pascal VOC-SP, COCO-SP, Petides-Struct, Petides-Func and PCQM-Contact from Long Range Graph Benchmark (LRGB; Dwivedi et al., 2022b), and the Tree Neighbour Match dataset (Alon & Yahav, 2021). The molecular graphs in ZINC range from 9 to 37 nodes, where each node represents a heavy atom (with 28 possible atom types) and each edge signifies a bond (with 3 possible types). The primary task is to regress a molecular property known as constrained solubility. The dataset includes a predefined train/validation/test split of 10K/1K/1K instances. Additionally, Appendix A mentions ZINC is sourced from "a freely available database of commercially accessible compounds (Irwin et al., 2012)". |
| Dataset Splits | Yes | The classification tasks involve the standard dataset splits of 55K/5K/10K for MNIST and 45K/5K/10K for CIFAR10, corresponding to train/validation/test graphs. The dataset includes a predefined train/validation/test split of 10K/1K/1K instances. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory specifications) used for running the experiments. |
| Software Dependencies | No | The paper mentions using the Adam W optimizer but does not specify version numbers for any software dependencies or libraries (e.g., Python, PyTorch, CUDA). |
| Experiment Setup | Yes | Table 6. Hyperparameters used for GEAET on 6 datasets. Hyperparameter CIFAR10 MNIST PATTERN Peptides-Struct Pascal VOC-SP COCO-SP Layers 5 5 7 6 8 8 Hidden Dim d 40 40 64 224 68 68 MPNN Gated GCN Gated GCN Gated GCN GCN Gated GCN Gated GCN Self Attention Transformer Transformer Transformer None Transformer Transformer External Network GEANet GEANet GEANet GEANet GEANet GEANet Self Heads 4 4 4 None 4 4 External Heads 4 4 4 8 4 4 Unit Size S 10 10 16 28 17 17 PE ESLap PE-8 ESLap PE-8 RWPE-16 Lap PE-10 None None PE Dim 8 8 7 16 None None Batch Size 16 16 32 200 50 50 Learning Rate 0.001 0.001 0.0005 0.001 0.001 0.001 Num Epochs 150 150 100 250 200 200 Warmup Epochs 5 5 5 5 10 10 Weight Decay 1e-5 1e-5 1e-5 0 0 0 Num Parameters 113,235 113,155 429,052 463,211 506,213 505,661 |