Instance Enhancement Batch Normalization: An Adaptive Regulator of Batch Noise
Authors: Senwei Liang, Zhongzhan Huang, Mingfu Liang, Haizhao Yang4819-4827
AAAI 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In our experiments, we replace all the BNs in the original networks with IEBN. The implementation details can be found in Appendix. Image Classification. As shown in Table 1, IEBN improves the testing accuracy over BN for different datasets and different network backbones. For small-class-dataset CIFAR10, the performance of the networks with BN is good enough, so there is not large space for improvement. However, for CIFAR100 and Image Net, the networks with IEBN achieve a significant testing accuracy improvement over BN. In particular, the performance improvement of Res Net with the IEBN is most remarkable. Also, IEBN has comparable performance to SE but has the advantage over SE in the aspect of low parameter increment. Analysis In this session, we further explore the role of the self-attention mechanism on enhancing instance information and regulating the batch noise. We analyze through the experiments with two kinds of noise attack designed in our paper. Constant-Noise Attack We add the constant noise into every BN of the network at the batch-normalized step as followed, Mix-Dataset Attack In this part, we consider interfering with μB c and σB c by simultaneously training on the datasets with different distributions in one network. Ablation Study In this section, we conduct experiments to explore the effect of different configurations of IEBN. All experiments are performed on CIFAR100 with Res Net164 using 2 GPUs. |
| Researcher Affiliation | Academia | Senwei Liang,1 Zhongzhan Huang,2 Mingfu Liang,3 Haizhao Yang1,4 1Purdue University, 2Tsinghua University, 3Northwestern University, 4National University of Singapore |
| Pseudocode | Yes | Algorithm 1 The forward pass of IEBN during training Input: X is a batch input of size B C H W; Parameters: γc, βc, ˆγc and ˆβc, c = 1, , C. Output: {Y = IEBNγc,βc,ˆγc, ˆβc(X)}. |
| Open Source Code | No | The paper does not provide any information or link about open-source code for the methodology. |
| Open Datasets | Yes | We conduct experiments on CIFAR10, CIFAR100 (Krizhevsky and Hinton 2009), and Image Net 2012 (Russakovsky et al. 2015). CIAFR10 or CIFAR100 has 50k train images and 10k test images of size 32 by 32 but has 10 and 100 classes respectively. Image Net 2012 (Russakovsky et al. 2015) comprises 1.28 million images for training and 50k for validation from 1000 classes, and the random cropping of size 224 by 224 is used in our experiments. |
| Dataset Splits | Yes | CIAFR10 or CIFAR100 has 50k train images and 10k test images of size 32 by 32 but has 10 and 100 classes respectively. Image Net 2012 (Russakovsky et al. 2015) comprises 1.28 million images for training and 50k for validation from 1000 classes, and the random cropping of size 224 by 224 is used in our experiments. |
| Hardware Specification | Yes | S. Liang and H. Yang gratefully acknowledge ... the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research. Z. Huang thanks New Oriental AI Research Academy Beijing for GPU resources. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers. |
| Experiment Setup | Yes | The idea behind IEBN is simple. As shown in Fig. 1, IEBN extracts the instance statistic of a channel before BN and applies it to rescale the corresponding output channel of BN with a pair of additional parameters. ... The parameters ˆγc, ˆβc are initialized by constant 0 and -1 respectively. We will discuss it in Ablation Study. ... We conduct experiments on CIFAR10, CIFAR100 (Krizhevsky and Hinton 2009), and Image Net 2012 (Russakovsky et al. 2015). ... the random cropping of size 224 by 224 is used in our experiments. We evaluate our method on popular networks, including Res Net (He et al. 2016), Pre Res Net (He et al. 2016), Res Ne Xt (Xie et al. 2017), Dense Net (Huang et al. 2017). In our experiments, we replace all the BNs in the original networks with IEBN. ... The implementation details can be found in Appendix. |