Improving Breadth-Wise Backpropagation in Graph Neural Networks Helps Learning Long-Range Dependencies.
Authors: Denis Lukovnikov, Asja Fischer
ICML 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To analyse the consequences, we design synthetic datasets serving as a testbed for the ability of GNNs to learn long-range patterns. Our analysis shows that several commonly used GNN variants with only depth-wise skip connections indeed have problems learning long-range patterns. We also verify that the presented architecture is competitive on real-world data. Third, we propose two synthetic datasets that challenge the ability of GNNs in learning long-range patterns. We use these datasets to perform an empirical analysis of commonly used GNNs and the proposed architecture. In addition, we perform experiments on the ZINC and OGBG-CODE2 datasets to study real-world performance. |
| Researcher Affiliation | Academia | Denis Lukovnikov 1 Asja Fischer 1 1Ruhr University Bochum, Bochum, Germany. Correspondence to: Denis Lukovnikov <denis.lukovnikov@rub.de>, Asja Fischer <asja.fischer@rub.de>. |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code is available at https://github.com/ lukovnikov/resrgat. |
| Open Datasets | Yes | Third, we propose two synthetic datasets that challenge the ability of GNNs in learning long-range patterns. In addition, we perform experiments on the ZINC and OGBG-CODE2 datasets to study real-world performance. ZINC contains 10k training, 1k validation, and 1k test examples. We also run experiments on the OGBG-CODE2 dataset provided as part of the Open Graph Benchmark initiative (Hu et al., 2020). |
| Dataset Splits | Yes | Twenty examples were randomly generated per output class for a total of 1220 examples and the data was split in 80/10/10 train/validation/test splits. A 50/25/25% training/validation/test split is used. ZINC contains 10k training, 1k validation, and 1k test examples. OGBG-CODE2 s original project-based split contains 408k training, 23k validation, and 22k test examples. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory) used for running its experiments. |
| Software Dependencies | No | The paper mentions using the “DGL framework” but does not provide specific version numbers for any software dependencies. |
| Experiment Setup | Yes | We use the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.001 and a batch size of 32. The model is trained using the Adam optimizer with learning rate 0.001 for 100 epochs. In the 4-layer setup, weight sharing is not used. A partially weight-shared Res RGAT is used for the 16-layer experiments to reduce the total number of parameters while not using full weight sharing, i.e. eight unique layers are used, each repeated twice consecutively. We use 10 layers, where every two layers share weights, and mean pooling to build the final graph representation vector. |