SimCalib: Graph Neural Network Calibration Based on Similarity between Nodes
Authors: Boshi Tang, Zhiyong Wu, Xixin Wu, Qiaochu Huang, Jun Chen, Shun Lei, Helen Meng
AAAI 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimentally, we discover a correlation between nodewise similarity and model calibration improvement, in alignment with our theoretical results. Additionally, we conduct extensive experiments investigating different design factors and demonstrate the effectiveness of our proposed Sim Calib framework for GNN calibration by achieving state-of-the-art performance on 14 out of 16 benchmarks. |
| Researcher Affiliation | Academia | 1Shenzhen International Graduate School, Tsinghua University, Shenzhen, China 2The Chinese University of Hong Kong, Hong Kong SAR, China |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide a specific link or explicit statement about the public release of its source code. |
| Open Datasets | Yes | Cora (Mc Callum et al. 2000), Citeseer (Giles, Bollacker, and Lawrence 1998), Pubmed (Sen et al. 2008), Cora Full (Bojchevski and G unnemann 2017), and the four Amazon datasets (Shchur et al. 2018). |
| Dataset Splits | Yes | To reduce the influence of randomness, we randomly assign 15% of nodes as L, and the rest as U, and we repeat this assignment process with randomness five times for each dataset. Once L has been sampled, we use three-fold cross-validation on it. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory) used for running its experiments. |
| Software Dependencies | No | The paper mentions 'scipy (Virtanen et al. 2020)' but does not specify version numbers for other key software components or libraries required for reproduction. |
| Experiment Setup | No | We train calibrators on validation sets by minimizing NLL loss, and validate it on the training set, following the common practice (Wang et al. 2021). We provide details of comparison settings and hyperparameters in Appendix. |