GOAT: A Global Transformer on Large-scale Graphs
Authors: Kezhi Kong, Jiuhai Chen, John Kirchenbauer, Renkun Ni, C. Bayan Bruss, Tom Goldstein
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the competitiveness of GOAT on both heterophilious and homophilious graphs with millions of nodes. |
| Researcher Affiliation | Collaboration | 1University of Maryland, College Park 2Capital One. Correspondence to: Kezhi Kong <kong@cs.umd.edu>, Tom Goldstein <tomg@cs.umd.edu>. |
| Pseudocode | Yes | Algorithm 1 Global Transformer mini-batch forward propogation algorithm; Algorithm 2 EMA K-Means update algorithm |
| Open Source Code | Yes | We open source our implementation at https://github.com/devnkong/GOAT. |
| Open Datasets | Yes | We select four datasets to evaluate. To represent homophilious graph problems we choose two datasets, ogbn-arxiv and ogbn-products, from the well-known Open Graph Benchmark (OGB) (Hu et al., 2020a). For heterophilious examples we utilize the arxiv-year and snap-patents datasets curated by Lim et al. (2021). |
| Dataset Splits | Yes | For the train and validation splits, we use the official splits from OGB for both ogbn-arxiv and ogbn-products and for arxiv-year and snap-patents we follow the practice of Lim et al. (2021) and randomly sample the train and validation sets. As there are no official splits or train set ratios for these two datasets, we experiment with training sets that comprise 10%, 20%, and 50% of the data while fixing validation set ratio at 25%, and report separate results for each split. |
| Hardware Specification | Yes | Each experiment is carried out on either a single Ge Force RTX 2080 Ti (11GB memory) or a RTX A4000 (16GB memory). |
| Software Dependencies | No | The paper mentions using Adam and AdamW optimizers, and that they utilize an implementation from another paper, but does not specify version numbers for Python, PyTorch, or other key software libraries used for the experiments. |
| Experiment Setup | Yes | For the local attention, we sample neighbors that live within 3-hops. For each node we sample [20, 10, 5] neighbors recursively. The size of the codebook is fixed at 4,096 and the dimensionality is 64. We always use a dropout rate of 0.5 and also use batch norm. We use Adam optimizer with lr 1e-3. |