NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
Authors: Arber Zela, Julien Siems, Frank Hutter
ICLR 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In order to allow a scientific study of these components, we introduce a general framework for one-shot NAS that can be instantiated to many recently-introduced variants and introduce a general benchmarking framework that draws on the recent large-scale tabular benchmark NAS-Bench-101 for cheap anytime evaluations of one-shot NAS methods. To showcase the framework, we compare several state-of-the-art one-shot NAS methods, examine how sensitive they are to their hyperparameters and how they can be improved by tuning their hyperparameters, and compare their performance to that of blackbox optimizers for NAS-Bench-101. |
| Researcher Affiliation | Collaboration | Arber Zela1 , Julien Siems1 , & Frank Hutter1,2 1Department of Computer Science, University of Freiburg 2Bosch Center for Artificial Intelligence |
| Pseudocode | Yes | Algorithm 1 DARTS ... Algorithm 2 PC-DARTS ... Algorithm 3 GDAS ... Algorithm 4 Random NAS with Weight-sharing ... Algorithm 5 ENAS |
| Open Source Code | Yes | We provide our open-source implementation1, which we expect will also facilitate the reproducibility and benchmarking of other one-shot NAS methods in the future. 1https://github.com/automl/nasbench-1shot1 |
| Open Datasets | Yes | NAS-Bench-101 (Ying et al., 2019) is a database of an exhaustive evaluation of all architectures in a constrained cell-structured space on CIFAR-10 (Krizhevsky, 2009). |
| Dataset Splits | Yes | The training dataset is split in two parts, one used for updating the parameters of the operations in the one-shot model, and the other to update the weights appended to operations, that determine the importance of that operation. ...each one-shot NAS optimizer will use 20k examples for training and 20k for search. |
| Hardware Specification | Yes | The search was done on a single NVIDIA RTX2080Ti using the same python environment. ...NAS-Bench-101, a large tabular benchmark ... using a one-time extreme amount of compute power (several months on thousands of TPUs) |
| Software Dependencies | No | The paper mentions 'python environment' and references various optimizers like DARTS and PC-DARTS, but it does not specify version numbers for Python or any specific libraries/frameworks (e.g., PyTorch, TensorFlow) used for the implementation. |
| Experiment Setup | Yes | We ran the NAS search for 50 epochs4 using their respective default hyperparameter settings (see Appendix C). ... We used a batch size of 96 throughout for DARTS, GDAS and PC-DARTS... Random WS was trained with a batch size of 64. |