NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension
Authors: Xin He, Jiangchao Yao, Yuxin Wang, Zhenheng Tang, Ka Chun Cheung, Simon See, Bo Han, Xiaowen Chu
AAAI 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments on NASBench-201 indicate that NAS-LID achieves superior performance with better efficiency. Specifically, compared to the gradient-driven method, NAS-LID can save up to 86% of GPU memory overhead when searching on NASBench-201. We also demonstrate the effectiveness of NAS-LID on Proxyless NAS and OFA spaces. |
| Researcher Affiliation | Collaboration | 1 Hong Kong Baptist University 2 Shanghai Jiao Tong University 3 Shanghai AI Laboratory 4 NVIDIA AI Tech Center 5 The Hong Kong University of Science and Technology (Guangzhou) 6 Mahindra University 7 Coventry University |
| Pseudocode | Yes | Alg. 1 details our LID-based supernet partition scheme. Algorithm 1: NAS-LID: LID-based Supernet Partition |
| Open Source Code | Yes | Source code: https://github.com/marsggbo/NAS-LID. |
| Open Datasets | Yes | NASBench-201 is a public tabular architecture dataset, which builds a DARTS-like (Liu, Simonyan, and Yang 2019) search space and provides the performance of 15,625 neural architectures on the CIFAR-10 and CIFAR-100 datasets (Krizhevsky and Hinton 2009). and We then transfer the searched architectures to the Image Net (Deng et al. 2009) dataset and present the results in Table 4. |
| Dataset Splits | Yes | NASBench-201 is a public tabular architecture dataset, which builds a DARTS-like (Liu, Simonyan, and Yang 2019) search space and provides the performance of 15,625 neural architectures on the CIFAR-10 and CIFAR-100 datasets (Krizhevsky and Hinton 2009). and We compare the ranking performance with RSPS (Li and Talwalkar 2020), GM-NAS (Hu et al. 2022) among the top 50/100/150 architectures in NASBench-201 space. and Fig. 5 compares the mean and standard deviation of validation accuracy of the searched architectures in each evolution epoch. |
| Hardware Specification | Yes | Table 1 compares the GPU memory (MB) costs between GM-NAS and our proposed NAS-LID on a single V100 GPU (32GB). |
| Software Dependencies | No | The paper does not explicitly list specific software dependencies with version numbers (e.g., programming languages, libraries, frameworks). |
| Experiment Setup | Yes | For the quantitative comparison, we first train the supernet for 50 epochs via Random Sampling with Parameter Sharing (RSPS) (Li and Talwalkar 2020) and use the pretrained weights to calculate the separability score for each edge, formalized as follows. and We finetune each sub-supernet for several epochs after partition. and For GM-NAS and our NAS-LID, we fine-tune the 16 sub-supernets for 50 epochs and then apply the evolutionary algorithm to search for superior architectures based on these sub-supernets. We conduct the evolutionary search for 50 epochs. and Input size Method NASBench-201 Proxyless NAS 128 3 32 32 and 32 3 224 224. |