NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
Authors: Xuanyi Dong, Yi Yang
ICLR 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this work, we propose an extension to NAS-Bench-101: NAS-Bench201 with a different search space, results on multiple datasets, and more diagnostic information. NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. The training log using the same setup and the performance for each architecture candidate are provided for three datasets. We have analyzed it from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability. |
| Researcher Affiliation | Collaboration | Xuanyi Dong and Yi Yang Re LER, CAI, University of Technology Sydney, Baidu Research |
| Pseudocode | No | The paper does not contain any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | Yes | We provide convenient APIs to access our NAS-Bench-201, which can be easily installed via pip install nas-bench-201. Please see https://github.com/D-X-Y/NAS-Bench-201 for more kinds of usages. |
| Open Datasets | Yes | We train and evaluate each architecture on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and Image Net-16-120 (Chrabaszcz et al., 2017). We choose these three datasets because CIFAR and Image Net (Russakovsky et al., 2015) are the most popular image classification datasets. |
| Dataset Splits | Yes | We split each dataset into training, validation and test sets to provide a consistent training and evaluation settings for previous NAS algorithms (Liu et al., 2019). CIFAR-10: We split all 50K training images in CIFAR-10 into two groups. Each group contains 25K images with 10 classes. We regard the first group as the new training set and the second group as the validation set. CIFAR-100: We randomly split the original test set into two group of equal size 5K images per group. One group is regarded as the validation set, and another one is regarded as the new test set. Image Net-16-120: In sum, Image Net-16-120 contains 151.7K training images, 3K validation images, and 3K test images with 120 classes. |
| Hardware Specification | Yes | We experimented all NAS algorithms on a single Ge Force GTX 1080 Ti GPU. The latency in our NAS-Bench-201 (version 1.0) is computed by running each model on a single GPU (Ge Force GTX 1080 Ti) with a batch size of 256. |
| Software Dependencies | No | The paper mentions implementing algorithms and using standard tools (like SGD, Adam), but it does not specify concrete version numbers for software dependencies (e.g., Python, PyTorch, TensorFlow versions, or specific library versions). |
| Experiment Setup | Yes | Table 1: The training hyper-parameter set H. optimizer SGD, initial LR 0.1, Nesterov, ending LR 0, momentum 0.9, LR schedule cosine, weight decay 0.0005, epoch 200, batch size 256, initial channel 16, V 4, N 5, random flip p=0.5, random crop normalization. |