Noise Injection Node Regularization for Robust Learning
Authors: Noam Itzhak Levi, Itay Mimouni Bloch, Marat Freytsis, Tomer Volansky
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We present theoretical and empirical evidence for substantial improvement in robustness against various test data perturbations for feed-forward DNNs when trained under NINR. |
| Researcher Affiliation | Academia | Noam Levi & Tomer Volansky Department of Physics Tel Aviv University Tel Aviv, Israel {noam,tomerv}@mail.tau.ac.il Itay M. Bloch Berkeley Center for Theoretical Physics, University of California and Theoretical Physics Group, Lawrence Berkeley National Laboratory, Berkeley, CA, U.S.A. itayblochm@berkeley.edu Marat Freytsis NHETC, Department of Physics and Astronomy Rutgers University Piscataway, NJ, U.S.A. marat.freytsis@rutgers.edu |
| Pseudocode | No | The paper does not contain any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | The models and tools used for analysis in our experiments are provided in the following anonymous link: https://anonymous.4open. science/r/Noise Injection Node Code-2A68 |
| Open Datasets | Yes | In the main text we present results mostly for the FMINST dataset (Xiao et al., 2017). These results also extend to more complex scenarios, demonstrated in similar experiments for the CIFAR-10 (Krizhevsky et al., 2014) dataset in App. E, while evidence for improvement against adversarial attacks is given in App. D as well as results for other noise distributions and optimizers beyond SGD. ... we consider the generalization between two different datasets, representing different marginal distributions, by training models with NINR on the MNIST dataset, and testing their performance on data drawn from a new target domain distribution: the USPS test set (Hull, 1994). ... The MNIST-C dataset (Mu & Gilmer, 2019) consists of 15 types of corruption applied to the MNIST test set, for benchmarking out-of-distribution robustness in computer vision. |
| Dataset Splits | Yes | FC and CNN architectures using the full dataset, consisting of 60 000 training examples with a 60/40 training/validation split. |
| Hardware Specification | No | The paper does not specify the hardware used for running the experiments, such as specific GPU or CPU models. |
| Software Dependencies | No | The paper mentions optimizers like SGD, RMSprop, and Adam, and implicitly uses a deep learning framework, but it does not specify versions for any software dependencies (e.g., Python, PyTorch, TensorFlow, specific libraries). |
| Experiment Setup | Yes | The learning rate is fixed to η = 0.05 with mini-batch size B = 128. Each training run is performed for 500 SGD training epochs in total, or until 98% training accuracy has been achieved. For further details, see Sec. 3 and App. A. ... The model parameters θ, WNI are initialized at iteration t = 0 using a normal distribution as σ2 θ0, σ2 WNI,0 = 1/dℓ, 1/dℓNI. |