Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
Authors: Alexandra Sasha Luccioni, Sylvain Viguier, Anne-Laure Ligozat
JMLR 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We estimate that BLOOM s final training emitted approximately 24.7 tonnes of CO2eq if we consider only the dynamic power consumption, and 50.5 tonnes if we account for all processes ranging from equipment manufacturing to energy-based operational consumption. We also carry out an empirical study to measure the energy requirements and carbon emissions of its deployment for inference via an API endpoint receiving user queries in real-time. |
| Researcher Affiliation | Collaboration | Alexandra Sasha Luccioni EMAIL Hugging Face Montréal, Canada Sylvain Viguier EMAIL Graphcore London, UK Anne-Laure Ligozat EMAIL LISN & ENSIIE Paris, France |
| Pseudocode | No | The paper describes the methodology for estimating carbon footprint but does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | All of the code and data used for our analyses are available in our Github repository. |
| Open Datasets | Yes | All of the code and data used for our analyses are available in our Github repository. |
| Dataset Splits | No | The paper estimates the carbon footprint of an existing model and its deployment, which does not involve traditional dataset splits for training, validation, or testing. |
| Hardware Specification | Yes | The BLOOM model was trained on HPE Apollo 6500 Gen10 Plus servers containing Nvidia A100 GPUs. [...] As reported in Table 1, training the BLOOM model required a total of 1.08 million GPU hours on a hardware partition constituted of Nvidia A100 SXM4 GPUs with 80GB of memory, which have a TDP of 400W (NVIDIA, 2022). [...] we ran the Code Carbon package (Schmidt et al., 2021) on a Google Cloud Platform (GCP) instance with 16 Nvidia A100 40GB GPUs, where BLOOM was deployed via an inference API... |
| Software Dependencies | No | The paper mentions using the 'Code Carbon package (Schmidt et al., 2021)' but does not provide a specific version number for this or any other software dependency. |
| Experiment Setup | Yes | Total training time 118 days, 5 hours, 41 min Total number of GPU hours 1,082,990 hours Total energy used 433,196 k Wh GPU models used Nvidia A100 80GB Carbon intensity of the energy grid 57 g CO2eq/k Wh... For deployment and inference: The model received an average of 558 requests per hour, which were handled in real time (i.e. without any batching), for 230,768 requests in total. ...the GCP instance used for deploying the BLOOM model is running in the us-central1 region, which has a carbon intensity of 394 g CO2eq/k Wh (Google, 2022) |