Speedy Performance Estimation for Neural Architecture Search
Authors: Robin Ru, Clare Lyle, Lisa Schut, Miroslav Fil, Mark van der Wilk, Yarin Gal
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate on various NAS search spaces that our estimator consistently outperforms other alternatives in achieving better correlation with the true test performance rankings. We further show that our estimator can be easily incorporated into both query-based and one-shot NAS methods to improve the speed or quality of the search. 4 Experiments In this section, we first evaluate the quality of our proposed estimators in predicting the generalisation performance of architectures against a number of baselines (Section 4.2), and then demonstrate that simple incorporation of our estimators can significantly improve the search speed and quality of both query-based and weight-sharing NAS (Sections 4.3 and 4.4). |
| Researcher Affiliation | Academia | 1 OATML Group, Department of Computer Science, University of Oxford, UK 2 Department of Computing, Imperial College London, UK |
| Pseudocode | No | The paper describes procedures and definitions mathematically but does not include any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our code is available at https://github.com/rubinxin/TSE. |
| Open Datasets | Yes | NASBench-201 (NB201) [11] 200 6466 15625 CIFAR10, CIFAR100, Image Net-16-120 |
| Dataset Splits | Yes | NASBench-201 (NB201) [11] 200 6466 15625 CIFAR10, CIFAR100, Image Net-16-120 |
| Hardware Specification | Yes | All experiments were conducted on an internal cluster of 16 RTX2080 GPUs. |
| Software Dependencies | No | The paper does not provide specific version numbers for software dependencies such as deep learning frameworks or programming languages. |
| Experiment Setup | Yes | To ensure fair assessment of the architecture performance only, we adopt the common NAS protocol where all architectures searched/compared are trained and evaluated under the same set of hyper-parameters. |