Temporal Generalization Estimation in Evolving Graphs
Authors: Bin Lu, Tingyan Ma, Xiaoying Gan, Xinbing Wang, Yunqiang Zhu, Chenghu Zhou, Shiyu Liang
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We theoretically establish a lower bound, proving that under mild conditions, representation distortion inevitably occurs over time. To estimate the temporal distortion without human annotation after deployment, one naive approach is to pre-train a recurrent model (e.g., RNN) before deployment and use this model afterwards, but the estimation is far from satisfactory. In this paper, we analyze the representation distortion from an information theory perspective, and attribute it primarily to inaccurate feature extraction during evolution. Consequently, we introduce SMART, a straightforward and effective baseline enhanced by an adaptive feature extractor through self-supervised graph reconstruction. In synthetic random graphs, we further refine the former lower bound to show the inevitable distortion over time and empirically observe that SMART achieves good estimation performance. Moreover, we observe that SMART consistently shows outstanding generalization estimation on four real-world evolving graphs. The ablation studies underscore the necessity of graph reconstruction. For example, on OGB-ar Xiv dataset, the estimation metric MAPE deteriorates from 2.19% to 8.00% without reconstruction. |
| Researcher Affiliation | Academia | 1Department of Electronic Engineering, Shanghai Jiao Tong University 2Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences 3John Hopcroft Center for Computer Science, Shanghai Jiao Tong University |
| Pseudocode | Yes | Algorithm 1 in Appendix A outlines the pre-deployment warm-up training and postdeployment finetuning process of SMART in detail. |
| Open Source Code | No | The paper does not provide a direct link or explicit statement about the release of their open-source code for the described methodology. It only references implementations from `networkx` for specific graph models in the Appendix. |
| Open Datasets | Yes | We use two citation datasets, a co-authorship network dataset and a series of social network datasets for evaluation. The statstics and more details are provided in Appendix F. OGBar Xiv (Hu et al., 2020a): A citation network between ar Xiv papers... DBLP (Galke et al., 2021): A citation network focused on computer science. ... Pharmabio (Galke et al., 2021): A co-authorship graph... Facebook 100 (Lim et al., 2021): A social network from Facebook of 5 different university: Penn, Amherst, Reed, Johns Hopkins and Cornell. |
| Dataset Splits | No | The paper describes training and testing phases and data splits for those, but it does not explicitly mention a separate 'validation' dataset split for its own model evaluation, nor specific percentages or counts for a validation set. |
| Hardware Specification | Yes | All the evaluated models are implemented on a server with two CPUs (Intel Xeon Platinum 8336C 2) and four GPUs (NVIDIA Ge Force RTX 4090 8). |
| Software Dependencies | No | The paper mentions using 'Adam optimizer' and 'Re LU activation function' and states that 'All the evaluated models are implemented on a server', but does not list specific software versions (e.g., Python, PyTorch, TensorFlow) or library versions used for implementation. |
| Experiment Setup | Yes | We use Adam optimizer for all the experiments, and the learning rate for all datasets are uniformly set to be 1e-3. In all experiments, the pre-trained graph neural networks are equipped with batch normalization and residual connections, with a dropout rate set to 0.1. Meanwhile, We employed the Re LU activation function. We set hyperparameter for each datasets and specify the details in Table 8. |