Hierarchical Representations for Efficient Architecture Search
Authors: Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, Koray Kavukcuoglu
ICLR 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to Image Net, which is competitive with the best existing neural architecture search approaches. We also present results using random search, achieving 0.3% less top-1 accuracy on CIFAR-10 and 0.1% less on Image Net whilst reducing the search time from 36 hours down to 1 hour. |
| Researcher Affiliation | Collaboration | Hanxiao Liu Carnegie Mellon University hanxiaol@cs.cmu.edu Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, Koray Kavukcuoglu Deep Mind {simonyan,vinyals,chrisantha,korayk}@google.com |
| Pseudocode | Yes | Algorithm 1: ASYNCEVO Asynchronous Evolution (Controller) Input: Data queue Q containing initial genotypes; Memory table M recording evaluated genotypes and their fitness. and Algorithm 2: ASYNCEVO Asynchronous Evolution (Worker) Input: Training set T , validation set V; Shared memory table M and data queue Q. |
| Open Source Code | No | Information insufficient. The paper does not state that the source code for their methodology is released or provide a link to it. |
| Open Datasets | Yes | Architecture search is carried out entirely on the CIFAR-10 training set, which we split into two sub-sets of 40K training and 10K validation images. Candidate models are trained on the training subset, and evaluated on the validation subset to obtain the fitness. Once the search process is over, the selected cell is plugged into a large model which is trained on the combination of training and validation sub-sets, and the accuracy is reported on the CIFAR-10 test set. We note that the test set is never used for model selection, and it is only used for final model evaluation. We also evaluate the cells, learned on CIFAR-10, in a large-scale setting on the Image Net challenge dataset (Sect. 4.3). |
| Dataset Splits | Yes | Architecture search is carried out entirely on the CIFAR-10 training set, which we split into two sub-sets of 40K training and 10K validation images. |
| Hardware Specification | Yes | it takes 1 hour to compute the fitness of one architecture on a single P100 GPU (which involves 4 rounds of training and evaluation). Using 200 GPUs, it thus takes 1 hour to perform random search over 200 architectures and 1.5 days to do the evolutionary search with 7000 steps. |
| Software Dependencies | No | Information insufficient. The paper mentions standard deep learning components (e.g., SGD, batch normalization, ReLU) but does not provide specific version numbers for any software or libraries used. |
| Experiment Setup | Yes | For fitness computation we use a smaller model with c0 = 16 and N = 1, shown in Fig. 2 (top-left). It is trained using SGD with 0.9 momentum for 5000 steps, starting with the learning rate 0.1, which is reduced by 10x after 4000 and 4500 steps. The batch size is 256, and the weight decay value is 3 10 4. We employ standard training data augmentation where a 24 24 crop is randomly sampled from a 32 32 image, followed by random horizontal flipping. The evaluation is performed on the full size 32 32 image. |