SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS
Authors: Yameng Peng, Andy Song, Haytham M. Fayek, Vic Ciesielski, Xiaojun Chang
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Comprehensive experiments validate its robust generalisation and superior performance across five benchmark search spaces, i.e., stack-based and cell-based, and seven tasks, i.e., image classification, object detection, autoencoding and jigsaw puzzle, outperforming 15 existing training-free metrics including recent proposed NWOT and Zi Co. |
| Researcher Affiliation | Academia | Yameng Peng Andy Song Haytham M. Fayek Vic Ciesielski Xiaojun Chang ,ξ School of Computing Technologies, RMIT University, Australia University of Technology Sydney, ξMohamed bin Zayed University of Artificial Intelligence |
| Pseudocode | Yes | Algorithm 1 SWAP-NAS |
| Open Source Code | Yes | 1Our code is available at https://github.com/pym1024/SWAP. |
| Open Datasets | Yes | Image classification tasks: CIFAR-10 / CIFAR-100 (Krizhevsky, 2009), Image Net-1k (Deng et al., 2009) and Image Net16-120 (Chrabaszcz et al., 2017). |
| Dataset Splits | Yes | For example, Spearman s rank correlation coefficient between regularised SWAP-Score and CIFAR-100 validation accuracies on NAS-Bench-201 networks is 0.90, significantly higher than 0.80 from the second-best metric, NWOT. These extensive studies follow the same setup as NAS-Bench-Suite-Zero (Krishnakumar et al., 2022), which is a standardised framework for verifying the effectiveness of training-free metrics. |
| Hardware Specification | Yes | All experiments are conducted on a single Tesla V100 GPU. |
| Software Dependencies | No | The paper does not provide specific version numbers for software dependencies or libraries used in the experiments. |
| Experiment Setup | Yes | The hyper-parameters, such as batch size, input data, sampled architectures and random seeds are fixed and consistently applied to all training-free metrics as NAS-Bench-Suite-Zero. The networks training strategy and hyper-parameters are exactly following the setup in DARTS (Liu et al., 2019). SWAP-NAS-A (µ=0.9, σ=0.9), SWAP-NAS-B (µ=1.2, σ=1.2), SWAP-NAS-C (µ=1.5, σ=1.5). SWAP-NAS (µ=25, σ=25). |