AutoTransfer: AutoML with Knowledge Transfer - An Application to Graph Neural Networks
Authors: Kaidi Cao, Jiaxuan You, Jiaju Liu, Jure Leskovec
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate AUTOTRANSFER on six datasets in the graph machine learning domain. Experiments demonstrate that (i) our proposed task embedding can be computed efficiently, and that tasks with similar embeddings have similar best-performing architectures; (ii) AUTOTRANSFER significantly improves search efficiency with the transferred design priors, reducing the number of explored architectures by an order of magnitude. |
| Researcher Affiliation | Academia | Department of Computer Science, Stanford University {kaidicao, jiaxuan, jiajuliu, jure}@cs.stanford.edu |
| Pseudocode | Yes | Algorithm 1 Summary of AUTOTRANSFER search pipeline; Algorithm 2 Training Pipeline for the projection function g( ) |
| Open Source Code | Yes | Source code is available at https://github.com/snap-stanford/AutoTransfer. |
| Open Datasets | Yes | Finally, we release GNN-BANK-101 the first large-scale database containing detailed performance records for 120,000 task-model combinations which were trained with 16,128 GPU hours to facilitate future research. |
| Dataset Splits | Yes | Coauthor Physics and Cora Full are transductive node classification datasets, so we randomly assign nodes into train/valid/test sets following a 50%:25%:25% split (Qin et al., 2021). We randomly split graphs following a 80%:10%:10% split for the three graph classification datasets (Qin et al., 2021). We follow the default train/valid/test split for the OGB-Arxiv dataset (Hu et al., 2020). |
| Hardware Specification | Yes | In our experiments, it typically takes a few seconds on an NVIDIA T4 GPU. |
| Software Dependencies | No | The paper mentions that the codebase was developed based on Graph Gym, but it does not specify version numbers for any software components or libraries, such as Python, PyTorch, or specific GNN frameworks. |
| Experiment Setup | Yes | Our GNN model specifications are summarized in Table 3. Our code base was developed based on Graph Gym (You et al., 2020). For all the training trials, We use the Adam optimizer and cosine learning rate scheduler (annealed to 0, no restarting). We use L2 regularization with a weight decay of 5e-4. ... We use the Adam optimizer with a learning rate of 5e-3. We use margin = 0.1 and train the network for 1000 iterations with a batch size of 128. ... Table 3: Design choices in our search space ... Learning rate 0.1, 0.001 Training epochs 200, 800, 1600. |