COPAL: Continual Pruning in Large Language Generative Models
Authors: Srikanth Malla, Joon Hee Choi, Chiho Choi
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our empirical evaluation on a various size of LLMs show that COPAL outperforms baseline models, demonstrating its efficacy in efficiency and adaptability. |
| Researcher Affiliation | Industry | 1Samsung Semiconductor, San Jose, USA. Correspondence to: Joon Hee Choi <jh4.choi@samsung.com>, Chiho Choi <chiho1.choi@samsung.com>. |
| Pseudocode | Yes | Algorithm 1 COPAL |
| Open Source Code | No | The paper does not include an explicit statement about releasing the source code or a direct link to a code repository for the methodology described. |
| Open Datasets | Yes | Our experimentation focused on three of the most commonly used language datasets in the field: Wikitext-2 (Merity et al., 2016), the Penn Treebank (PTB) (Marcus et al., 1993), and the Colossal Clean Crawled Corpus (C4) (Raffel et al., 2020). |
| Dataset Splits | No | The paper states using well-known datasets (Wikitext-2, PTB, C4) and defines calibration data for pruning, but it does not provide explicit training, validation, or test dataset splits (e.g., percentages or counts) for the overall experiment or model evaluation. |
| Hardware Specification | Yes | Our pruning experiments utilize a single NVIDIA A100 GPU with 80 GB of memory. |
| Software Dependencies | No | The paper mentions using the 'Pytorch framework (Paszke et al., 2019)' and the 'Hugging Face Transformers library (Wolf et al., 2019)', but does not provide specific version numbers for these software dependencies. |
| Experiment Setup | Yes | For calibration, we use 16 segments of 2048 tokens each, randomly chosen from the first shard of each dataset. Our experiments are conducted in a single step, without fine-tuning... Uniform sparsity is maintained across all linear layers for consistency. We explore three distinct sparsity types: unstructured sparsity, and semistructured sparsities with 4:8 and 2:4 configurations. |