LASER: Linear Compression in Wireless Distributed Optimization
Authors: Ashok Vardhan Makkuva, Marco Bondaschi, Thijs Vogels, Martin Jaggi, Hyeji Kim, Michael Gastpar
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically demonstrate the superiority of LASER over the baselines on the challenging tasks of (i) language modeling with GPT-2 WIKITEXT-103 and (ii, iii, iv) image classification on MNIST, CIFAR10 and CIFAR100. With high gradient compression (165 ), LASER achieves 5064% perplexity improvement in the low and moderate power regimes on WIKITEXT-103. |
| Researcher Affiliation | Academia | Ashok Vardhan Makkuva * 1 Marco Bondaschi * 1 Thijs Vogels 1 Martin Jaggi 1 Hyeji Kim 2 Michael Gastpar 1... 1School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland 2Department of Electrical and Computer Engineering, UT Austin, Austin, TX, USA. |
| Pseudocode | Yes | Algorithm 1 LASER |
| Open Source Code | Yes | Code is available at https: //github.com/Bond1995/LASER. |
| Open Datasets | Yes | We empirically demonstrate the superiority of LASER over the baselines on the challenging tasks of (i) language modeling with GPT-2 WIKITEXT-103 and (ii, iii, iv) image classification on MNIST, CIFAR10 and CIFAR100. |
| Dataset Splits | No | The paper uses standard benchmark datasets like WIKITEXT-103, CIFAR10, CIFAR100, and MNIST, but does not explicitly state the training, validation, and test splits (e.g., percentages or exact sample counts) within the provided text. |
| Hardware Specification | No | No specific hardware details such as GPU models, CPU models, or memory specifications were mentioned for running the experiments. |
| Software Dependencies | No | The paper mentions models and optimizers like 'GPT-2 like architecture' and 'Adam W', but does not provide specific version numbers for software dependencies or libraries (e.g., PyTorch, TensorFlow, Python versions). |
| Experiment Setup | Yes | Table 6: Default experimental settings for the GPT-2 model used to learn the WIKITEXT-103 task. ... Table 8: Default experimental settings for the RESNET18 model used to learn the CIFAR10 task. |