Plug-in, Trainable Gate for Streamlining Arbitrary Neural Networks
Authors: Jaedeok Kim, Chiyoun Park, Hyun-Joo Jung, Yoonsuck Choe4452-4459
AAAI 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experimental results demonstrate that the proposed method efficiently optimizes arbitrary neural networks in various tasks such as image classification, style transfer, optical flow estimation, and neural machine translation. |
| Researcher Affiliation | Collaboration | Jaedeok Kim,1 Chiyoun Park,1 Hyun-Joo Jung,1 Yoonsuck Choe1,2 1Artificial Intelligence Center, Samsung Research, Samsung Electronics Co. 56 Seongchon-gil, Secho-gu, Seoul, Korea, 06765 2Department of Computer Science and Engineering, Texas A&M University College Station, TX, 77843, USA |
| Pseudocode | No | No structured pseudocode or algorithm blocks were found in the paper. |
| Open Source Code | No | The paper mentions external source code used for specific tasks (e.g., "Source code: https://github.com/tensorflow/magenta/tree/master/magenta/models/image_stylization" and "Source code: https://github.com/sampepose/flownet2-tf") but does not provide a link or statement for the authors' own implementation of the proposed methodology. |
| Open Datasets | Yes | We used CIFAR-10 and Image Net datasets for our image classification experiments. |
| Dataset Splits | Yes | The performance of model is measured in average end-point-error (EPE) of validation data. |
| Hardware Specification | Yes | For inference time, we ran both model on Intel Core i7-7700@3.60GHz CPU. |
| Software Dependencies | No | The paper mentions software like Keras and fairseq but does not provide specific version numbers for these or other software dependencies. |
| Experiment Setup | Yes | In training phase, we used Image Net as a set of content images and chose 5 style images (i.e., N = 5) manually as shown in Figure 6. We trained both original and compressed model from scratch for 20K iterations with a batch size of 16. The number of pruned channels is used as a regularization factor with regularization weight λ = 0.1. The compressed model (ρch = 0.1) is 34.5 times smaller than the original model in terms of file size (Table 2). In order to see the actual inference time, we measured the inference time on the CPU of a Galaxy S10. The compressed model (ρch = 0.1) is more than 3 times faster in terms of the inference time as shown although the generation quality preserved as shown in Figure 6. Figure 7 shows the number of retained channels in each layer. The TGL does not select the number of pruned channels uniformly and it automatically selects which channels are pruned according to the objective function. Without the further pruning steps, our proposed method can train and prune simultaneously the model with a consideration of the target compression ratio as we mentioned in the previous section. Optical Flow Estimation We next consider a task that learns the optical flow between images (Dosovitskiy et al. 2015; Ilg et al. 2017). In this experimentation, we used Flow Net Simple (Dosovitskiy et al. 2015), which is the same as Flow Net S3 in (Ilg et al. 2017). Flow Net S stacks two consecutive input images together and feed them into the network to extract motion information between these images. Starting from the pre-trained model, a TGL is plugged into the output of every convolution and deconvolution layers except the last one for preserving its generation performance. We trained the model with TGLs for 1.2M iterations with a batch size of 8. The Adam optimizer (Kingma and Ba 2015) was used with initial learning rate 0.0001 and it was halved every 200K iterations after the first 400K iterations. |