NAS-Bench-x11 and the Power of Learning Curves
Authors: Shen Yan, Colin White, Yash Savani, Frank Hutter
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the power of using the full training information by introducing a learning curve extrapolation framework to modify single-fidelity algorithms, showing that it leads to improvements over popular single-fidelity algorithms which claimed to be state-of-the-art upon release. Our code and pretrained models are available at https://github.com/automl/nas-bench-x11. |
| Researcher Affiliation | Collaboration | Shen Yan 1, Colin White 2, Yash Savani3, Frank Hutter4,5 1 Michigan State University, 2 Abacus.AI, 3 Carnegie Mellon University, 4 University of Freiburg, 5 Bosch Center for Artificial Intelligence |
| Pseudocode | Yes | Algorithm 1 Single-Fidelity Algorithm |
| Open Source Code | Yes | Our code and pretrained models are available at https://github.com/automl/nas-bench-x11. |
| Open Datasets | Yes | NAS-Bench-101 [76], a tabular NAS benchmark, was created by defining a search space of size 423 624 unique architectures and then training all architectures from the search space on CIFAR-10 until 108 epochs. |
| Dataset Splits | Yes | We assume that we are given two datasets, Dtrain and Dtest, of architecture and learning curve pairs. We use Dtrain (often size > 10 000) to train the surrogate, and we use Dtest for evaluation. |
| Hardware Specification | No | The paper does not explicitly state specific hardware details (e.g., GPU/CPU models, memory) used for running its experiments. It mentions 'compute time and resources used in Section 5' in the ethics checklist, but Section 5 only discusses wall-clock time without hardware specifics. |
| Software Dependencies | No | The paper mentions software like 'LGBoost', 'XGBoost', 'MLPs', 'weighted probabilistic modeling (WPM)', and 'learning curve support vector regressor (Lc SVR)' but does not provide specific version numbers for these software dependencies in the main text. |
| Experiment Setup | Yes | Experimental setup. For each search space, we run each algorithm for a total wall-clock time that is equivalent to running 500 iterations of the single-fidelity algorithms for NAS-Bench-111 and NAS-Bench-311, and 100 iterations for NAS-Bench-201 and NAS-Bench-NLP11. |