Zero-Cost Operation Scoring in Differentiable Architecture Search
Authors: Lichuan Xiang, Lukasz Dudziak, Mohamed S. Abdelfattah, Thomas Chau, Nicholas D. Lane, Hongkai Wen
AAAI 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct a thorough empirical evaluation on a number of NAS benchmarks and large search spaces, from NAS-Bench-201, NAS-Bench-1Shot1, NASBench-Macro, to DARTS-like and Mobile Net-like spaces, showing significant improvements in both search time and accuracy. |
| Researcher Affiliation | Collaboration | 1 University of Warwick, UK 2 Samsung AI Center Cambridge, UK 3 Cornell University, USA 4 University of Cambridge, UK |
| Pseudocode | Yes | Our algorithm contains two stages: architecture proposal and validation. It begins with an untrained supernet A0 which contains a set of edges E, the number of proposal iterations N, and the number of validation iterations V. In each proposal iteration i, we discretize the supernet A0 based on our proposed zero-cost-based perturbation function fzc-pt that achieved promising results in the previous section. After all edges have been discretized, the final architecture is added to the set of candidates and we begin the process again for i + 1 starting with the original A0. After N candidate architectures have been constructed, the validation stage begins. We score the candidate architectures again using a selected zero-cost metric (the same which is used in fzc-pt), but this time computing their end-to-end score rather than using the perturbation paradigm. We calculate the zero-cost metric for each subnetwork using V different minibatches of data. The final architecture is selected by the best total score during the validation stage. The full algorithm is outlined as Algorithm 1 in Appendix A.5 and the flowchart of our algorithms is in Figure 3 . |
| Open Source Code | Yes | Our code is available at: https://github.com/zerocostptnas/zerocost operation score. |
| Open Datasets | Yes | We conduct experiments on two popular NAS benchmarks: NAS-Bench-201 (Dong and Yang 2020) and NAS-Bench-1Shot1 (Zela, Siems, and Hutter 2020). ... NAS-Bench-201 constructed a unified cell-based search space, where each architecture has been trained on three different datasets, CIFAR10, CIFAR-100 and Image Net-16-1202. |
| Dataset Splits | Yes | All searches are performed on CIFAR-10, and obtained architectures are then additionally evaluated on the other two datasets. We search with four random seeds (0, 1, 2, 3) and report the average and standard deviation of test errors of the obtained architectures. ... We use the three random seeds available in NAS-Bench-201: 777, 888, 999. |
| Hardware Specification | Yes | total searching time 25 minutes on a single GPU ... (2500 sample size, 25min on a single 2080ti GPU) |
| Software Dependencies | No | The paper mentions various zero-cost proxies and methods like 'grad norm, snip, grasp, synflow, fisher and nwot', and 'TE-NAS', but does not provide specific version numbers for any software dependencies or libraries used in the experiments. |
| Experiment Setup | Yes | We search with four random seeds (0, 1, 2, 3) and report the average and standard deviation of test errors of the obtained architectures. ... We study the impact of different architecture proposal iterations N and validation iterations V when Zero-Cost-PT uses nwot metric and random edge discretization order. ... (3) retrain for 5 epochs (darts-pt and disc-acc only) |