Band-limited Training and Inference for Convolutional Neural Networks
Authors: Adam Dziedzic, John Paparrizos, Sanjay Krishnan, Aaron Elmore, Michael Franklin
ICML 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimentally, we observe that Convolutional Neural Networks (CNNs) are resilient to this compression scheme and results suggest that CNNs learn to leverage lower-frequency components. In particular, we found: (1) band-limited training can effectively control the resource usage (GPU and memory); (2) models trained with band-limited layers retain high prediction accuracy; and (3) requires no modification to existing training algorithms or neural network architectures to use unlike other compression schemes. |
| Researcher Affiliation | Academia | 1Department of Computer Science, University of Chicago, Chicago, USA. Correspondence to: Adam Dziedzic <ady@uchicago.edu>, John Paparrizos <jopa@uchicago.edu>. |
| Pseudocode | No | No pseudocode or algorithm blocks were found in the paper. |
| Open Source Code | No | The paper does not provide an explicit statement about open-sourcing the code for the described methodology or a link to a code repository. |
| Open Datasets | Yes | Res Net-18 and Dense Net-121, on CIFAR-10 and CIFAR100 datasets, respectively. We used the Friedman statistical test (Friedman, 1937) followed by the post-hoc Nemenyi test (Nemenyi, 1962) to assess the performance of multiple compression rates during inference over multiple datasets (see supplementary material for details). Our results suggest that the best test accuracy is achieved when the same compression rate is used during training and inference and, importantly, the difference in accuracy is statistically significantly better in comparison to the test accuracy achieved with different compression rate during inference. |
| Dataset Splits | No | While the paper mentions training and testing on datasets like CIFAR-10 and CIFAR-100, it does not explicitly state the specific train/validation/test splits used (e.g., percentages, sample counts, or references to predefined splits with citations). |
| Hardware Specification | Yes | We run our experiments on single GPU deployments with NVidia P-100 GPUs and 16 GBs of total memory. |
| Software Dependencies | No | The paper mentions software like PyTorch, C++, CUDA, cuDNN, and Foolbox, but it does not specify version numbers for any of these components, which are necessary for reproducible software dependency information. |
| Experiment Setup | Yes | Specifically, we vary the compression rate between 0% and 50% for each convolutional layer (i.e., the percentage of coefficients discarded) and we train the two models for 350 epochs. We run Res Net-18 on CIFAR-10 with batch size 32 and we query the VBIOS (via nvidia-smi every millisecond in the window of 7 seconds). |