SWALP : Stochastic Weight Averaging in Low Precision Training
Authors: Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, Chris De Sa
ICML 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, we demonstrate that training with 8-bit SWALP can match the full precision SGD baseline in deep learning tasks such as training Preactivation Res Net-164 (He et al., 2016) on CIFAR-10 and CIFAR-100 datasets (Krizhevsky & Hinton, 2009). |
| Researcher Affiliation | Academia | 1Cornell University. Correspondence to: Guandao Yang <gy46@cornell.edu>, Tianyi Zhang <tz58@cornell.edu>. |
| Pseudocode | Yes | Algorithm 1 SWALP and Algorithm 2 SWALP with all numbers quantized. |
| Open Source Code | Yes | We provide code at https://github.com/stevenygd/SWALP. |
| Open Datasets | Yes | We use the CIFAR (Krizhevsky & Hinton, 2009) and Image Net (Russakovsky et al., 2014) datasets for our experiments. To empirically validate Theorem 2, we use logistic regression with L2 regularization on the MNIST dataset (Le Cun et al., 1998). |
| Dataset Splits | Yes | We use the CIFAR (Krizhevsky & Hinton, 2009) and Image Net (Russakovsky et al., 2014) datasets for our experiments. Following prior work (Izmailov et al., 2018a; Wu et al., 2018), we apply standard preprocessing and data augmentation for experiments on CIFAR datasets. SWALP s hyperparameters are obtained from grid search on a validation set. |
| Hardware Specification | No | The paper mentions “Google Cloud Platform Research Credits program for providing computational resources” but does not specify any particular GPU, CPU, or TPU models, or other detailed hardware specifications used for experiments. |
| Software Dependencies | No | The paper mentions adapting preprocessing and data augmentation from “the public Py Torch example (Paszke et al., 2017)” but does not specify version numbers for PyTorch or any other software dependencies. |
| Experiment Setup | No | The paper states “SWALP s hyperparameters are obtained from grid search on a validation set. Please see the Appendix for more detail.” and “Please see Appendix for the details on hyper-parameters.”, indicating that specific hyperparameter values are not provided in the main text. |