A Neural Collapse Perspective on Feature Evolution in Graph Neural Networks
Authors: Vignesh Kothapalli, Tom Tirer, Joan Bruna
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We start with an empirical study that shows that a decrease in within-class variability is also prevalent in the node-wise classification setting, however, not to the extent observed in the instance-wise case. Then, we theoretically study this distinction. ... Our main contributions can be summarized as follows: We conduct an extensive empirical study that shows that a decrease in within-class variability is prevalent also in the deepest features of GNNs trained for node classification on SBMs. ... In this section, we explore the evolution of the deepest features of GNNs during training. In Section 3.1, we present empirical results of GNNs in the setup that is detailed in Section 2, showing that a decrease in within-class feature variability is present in GNNs that reach zero training error, but not to the extent observed with plain DNNs. |
| Researcher Affiliation | Academia | Vignesh Kothapalli New York University Tom Tirer Bar-Ilan University Joan Bruna New York University |
| Pseudocode | No | The paper describes methods like GNN architectures and spectral clustering via mathematical formulations (equations 3, 4, 15, 16, 17) but does not include any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code is available at: https://github.com/kvignesh1420/gnn_collapse |
| Open Datasets | Yes | We employ the Symmetric Stochastic Block Model (SSBM) to generate graphs {Gk = (Vk, Ek)}K k=1. Stochastic block models (originated in [29]) are classical random graph models that have been extensively studied in statistics, physics, and computer science. In the SSBM model that is considered in this paper, each graph Gk is associated with an adjacency matrix Ak RN N, degree matrix Dk = diag(Ak1) RN N, and a random node features matrix Xk Rd N, with entries sampled from a normal distribution. Formally, if P RC C represents a symmetric matrix with diagonal entries p and off-diagonal entries q, a random graph Gk is considered to be drawn from the distribution SSBM(N, C, p, q) if an edge between vertices vi, vj is formed with probability (P)yk(vi),yk(vj)3. |
| Dataset Splits | No | No explicit mention of training/validation/test dataset splits (e.g., '80/10/10 split') or absolute sample counts for a validation set was found in the paper. The paper mentions training on K=1000 graphs and testing on K=100 graphs, but no dedicated validation set is specified. |
| Hardware Specification | Yes | We perform experiments on a virtual machine with 8 Intel(R) Xeon(R) Platinum 8268 CPUs, 32GB of RAM, and 1 Quadro RTX 8000 GPU with 32GB of allocated memory. |
| Software Dependencies | Yes | Our Python package gnn_collapse leverages Py Torch 1.12.1 and Py Torch-Geometric (Py G) 2.1.0 frameworks. |
| Experiment Setup | Yes | The networks ψF Θ, ψF Θ are composed of L = 32 layers with graph operator, Re LU activation, and instance-normalization functionality. The hidden feature dimension is set to 8 across layers. They are trained for 8 epochs using stochastic gradient descent (SGD) with a learning rate 0.004, momentum 0.9, and a weight decay of 5 10 4. |