CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework
Authors: Yiyang Zhao, Yunzhuo Liu, Bo Jiang, Tian Guo
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This work presents a novel approach to neural architecture search (NAS) that aims to increase carbon efficiency for the model design process. ... We demonstrate the efficacy of CE-NAS in lowering carbon emissions while achieving SOTA results for both NAS benchmarks and open-domain NAS tasks. For example, on the HW-Nas Bench, CE-NAS reduces carbon emissions by up to 7.22X while maintaining a search efficiency comparable to vanilla NAS. For opendomain NAS tasks, CE-NAS achieves SOTA results with 97.35% top-1 accuracy on CIFAR-10 with only 1.68M parameters and a carbon consumption of 38.53 lbs of CO2. |
| Researcher Affiliation | Academia | Yiyang Zhao Worcester Polytechnic Institute Yunzhuo Liu Shanghai Jiao Tong University Bo Jiang Shanghai Jiao Tong University Tian Guo Worcester Polytechnic Institute |
| Pseudocode | No | The paper provides a high-level overview of the CE-NAS framework in Figure 1 and describes its components and processes in detail within the text. However, it does not include any formal pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our code is available at https://github.com/cake-lab/CE-NAS. |
| Open Datasets | Yes | Our experiments utilize carbon trace data sourced from Electricity Map [58], an independent carbon information service. ... We align our training and testing data with the approach used in [55] by utilizing Electricity Map [58], a third-party carbon information service. We select three regions (namely, CISO6, DE7, and PJM8) from Electricity Map [58] to represent distinct carbon trace scenarios for our experiments. The dataset spans from January 1, 2020, to December 31, 2021, with an hourly resolution. |
| Dataset Splits | Yes | The dataset spans from January 1, 2020, to December 31, 2021, with an hourly resolution. We employ a train(validation)-test split of 75% 25%. |
| Hardware Specification | Yes | The batch size is configured at 512 on a single NVIDIA A100 GPU. ... To speed up the evaluation process, we use the NVIDIA Automatic Mixed Precision (AMP) library with FP16 for training during the search. ... For each architecture in the Pareto frontier, we train it on 8 Tesla V100 GPUs with a resolution of 320x320... |
| Software Dependencies | No | The paper mentions various software components and techniques such as 'NVIDIA Automatic Mixed Precision (AMP)', 'FP16', 'Tensor RT', 'Adam W optimizer', and 'SGD optimizer' but does not specify exact version numbers for any of these software dependencies. |
| Experiment Setup | Yes | Search setup: Initially, we utilize early stopping for training sampled architectures for 200 epochs, in contrast to 600 epochs required for the final training phase. To further hasten training, we reduce the initial channel size from 36 to 18 and increase the batch size to 320. Additionally, the number of layers is scaled down from 24 to 16 during the search phase. ... Training setup: The final selected architectures are trained for 600 epochs, using a batch size of 128 and a momentum SGD optimizer with an initial learning rate of 0.025. This rate is adjusted following a cosine learning rate schedule throughout the training. Weight decay is applied for regularization. |