Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning
Authors: Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, Bryan Hooi
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments demonstrate that our method achieves state-of-the-art results on well-established TAG datasets, including Cora, Pub Med, ogbn-arxiv, as well as our newly introduced dataset, tape-arxiv23. Furthermore, our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv. |
| Researcher Affiliation | Collaboration | Xiaoxin He1, Xavier Bresson1, Thomas Laurent2, Adam Perold3, Yann Le Cun4,5, Bryan Hooi1 1National University of Singapore, 2Loyola Marymount University 3Element, Inc., 4New York University, 5Meta AI |
| Pseudocode | No | The paper describes the proposed method in a step-by-step manner using text and equations, but it does not include a formal pseudocode block or algorithm listing. |
| Open Source Code | Yes | Our source code can be accessed at the following url: https://github.com/Xiaoxin He/TAPE. |
| Open Datasets | Yes | For Cora and Pub Med, raw text data of the articles is unavailable in common graph libraries such as Py G and DGL. Hence, we collected and formatted the missing text data for these datasets in TAG format. Additionally, given the popularity of these datasets, their TAG version will be released publicly for reproducibility and new research projects. For ogbn-products, given its substantial scale of 2 million nodes and 61 million edges and considering our academic resource budget, we conducted experiments on a subgraph sample. Details can be found in Appendix G. ... Furthermore, we introduce the new tape-arxiv23 citation graph dataset, extending beyond GPT-3 s knowledge cutoff, i.e., Sept. 2021. These datasets can serve as valuable resources for the NLP and GNN research community. |
| Dataset Splits | Yes | For the ogbn-arxiv and ogbn-products dataset, we adopted the standard train/validation/test split provided by OGB (Hu et al., 2020a). As for the Cora, Pub Med datasets, and tape-arxiv23, we performed the train/validation/test splits ourselves, where 60% of the data was allocated for training, 20% for validation, and 20% for testing. |
| Hardware Specification | Yes | LM-based experiments were performed on four NVIDIA RTX A5000 GPUs, each with 24GB VRAM. On the other hand, the GNN-based experiments were conducted on a single GPU. |
| Software Dependencies | No | The paper mentions using 'Py G and DGL modules' but does not specify their version numbers, nor does it list specific versions for other software dependencies like Python or PyTorch, which would be necessary for full reproducibility. |
| Experiment Setup | Yes | Table 10 provides an overview of the hyperparameters used for the GCN (Kipf & Welling, 2016), SAGE (Hamilton et al., 2017), and Rev GAT (Li et al., 2021) models. These hyperparameters were selected based on the official OGB repository and the Rev GAT and language model hyperparameters follow those used in the GLEM repository. It lists '# layers', 'hidden dim', 'learning rate', 'dropout', 'epoch', 'warmup epochs', and 'early stop'. |