NAS-Bench-101: Towards Reproducible Neural Architecture Search
Authors: Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, Frank Hutter
ICML 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the precomputed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms. |
| Researcher Affiliation | Collaboration | 1Google Brain, Mountain View, California, USA 2Department of Computer Science, University of Freiburg, Germany. |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Moreover, the data, search space, and training code is fully public 1, to foster reproducibility in the NAS community. 1 Data and code for NAS-Bench-101 available at https:// github.com/google-research/nasbench. The code, implemented in Tensor Flow, along with all chosen hyperparameters, is publicly available at https: //github.com/google-research/nasbench. |
| Open Datasets | Yes | We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. [...] We trained and evaluated a large number of different convolutional neural network (CNN) architectures on CIFAR-10 (Krizhevsky & Hinton, 2009) |
| Dataset Splits | Yes | All models are trained and evaluated on CIFAR-10 (40k training examples, 10k validation examples, 10k testing examples), using standard data augmentation techniques (He et al., 2016). |
| Hardware Specification | Yes | All models were trained on the TPU v2 accelerator. |
| Software Dependencies | No | The paper mentions "Tensor Flow" but does not provide a specific version number, nor does it list versions for any other software or libraries. |
| Experiment Setup | Yes | We utilize a single, fixed set of hyperparameters for all NAS-Bench-101 models. [...] All models are trained and evaluated on CIFAR-10 (40k training examples, 10k validation examples, 10k testing examples), using standard data augmentation techniques (He et al., 2016). The learning rate is annealed via cosine decay (Loshchilov & Hutter, 2017) to 0 in order to reduce the variance between multiple independent training runs. Training is performed via RMSProp (Tieleman & Hinton, 2012) on the cross-entropy loss with L2 weight decay. [...] we trained all our architectures with four increasing epoch budgets: Estop {Emax/33, Emax/32, Emax/3, Emax} = {4, 12, 36, 108} epochs. |