RSC: Accelerate Graph Neural Networks Training via Randomized Sparse Computations
Authors: Zirui Liu, Chen Shengyuan, Kaixiong Zhou, Daochen Zha, Xiao Huang, Xia Hu
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments have demonstrated the effectiveness of the proposed method. Particularly, RSC can achieve up to 11.6 speedup for a single sparse operation and a 1.6 end-to-end wall-clock time speedup with negligible ( 0.3%) accuracy drop. ... 6. Experiments: We verify the effectiveness of our proposed framework via answering the following research questions: Q1: How effective is RSC in terms of accuracy with reduced training time? Q2: How effective is our proposed allocation strategy compared to the uniform allocation strategy? Q3: What is the layer-wise ratio assigned by RSC ? Q4: How effective is the caching and switching mechanism in terms of the trade-off between efficiency and accuracy? ... Table 3: Comparison on the test accuracy/F1-micro/AUC and speedup on four datasets. |
| Researcher Affiliation | Academia | 1Department of Computer Science, Rice University, Houston, TX, USA 2Department of Computing, The Hong Kong Polytechnic University, Hung Hom, Hong Kong SAR. Correspondence to: Xia Hu <xia.hu@rice.edu>. |
| Pseudocode | Yes | We provide the pseudo-code of our greedy algorithm in Algorithm 1 of Appendix B. ... Algorithm 1 The greedy algorithm |
| Open Source Code | Yes | Codes are available at https://github.com/warai-0toko/RSC-ICML. |
| Open Datasets | Yes | To evaluate RSC , we adopt four common large-scale graph benchmarks from different domains, i.e., Reddit (Hamilton et al., 2017), Yelp (Zeng et al., 2020), ogbn-proteins (Hu et al., 2020), and ogbn-products (Hu et al., 2020). ... All datasets are directly downloaded from Pytorch Geometric or the protocol of OGB (Hu et al., 2020). |
| Dataset Splits | Yes | We follow the standard data splits and all datasets are directly downloaded from Pytorch Geometric or the protocol of OGB (Hu et al., 2020). ... We report the test accuracy associated with the highest validation score. |
| Hardware Specification | Yes | We measure the time on a single NVIDIA RTX3090 (24GB). The detailed software and hardware information can be found in Appendix D. ... All experiments are conducted on a server with four NVIDIA 3090 GPUs, four AMD EPYC 7282 CPUs, and 252GB host memory. |
| Software Dependencies | Yes | Here we list the details of our used packages in all experiments in Table 5. Table 5: Package configurations of our experiments. Package Version CUDA 11.1 pytorch sparse 0.6.12 pytorch scatter 2.0.8 pytorch geometric 1.7.2 pytorch 1.9.0 OGB 1.3.2 |
| Experiment Setup | Yes | Hyperparameter settings. RSC contains three parts. First, the allocation strategy. We choose the overall budget C in Equation (4b) from {0.1, 0.3, 0.5}. We run the resource allocation strategy every ten steps. The step size α in Algorithm 1 is set as 0.02|V|. Second, the caching mechanism. According to Figure 4, we sample the adjacency matrix every ten steps and reuse the sampled matrices for nearby steps. Third, the switching mechanism, where we apply RSC for 80% of the total epochs, while switching back to the original operations for the rest of the 20% epochs. ... Table 7: Configuration of Full-Batch GCN. ... Table 8: Configuration of Full-Batch Graph SAGE. ... Table 9: Configuration of Full-Batch GCNII. ... Table 10: Training configuration of Graph SAINT. |