The Effects of Regularization and Data Augmentation are Class Dependent
Authors: Randall Balestriero, Leon Bottou, Yann LeCun
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this study, we demonstrate that the optimal regularization s hyper-parameters found from cross-validation over all classes leads to disastrous model performances on a minority of classes. For example, a resnet50 trained on Imagenet sees its barn spider test accuracy falls from 68% to 46% only by introducing random crop DA during training. Our only theoretical result consists in a formal statistical test for which we precisely describe our settings and statistics. |
| Researcher Affiliation | Collaboration | Randall Balestriero Meta AI Research, FAIR NYC, USA rbalestriero@meta.com Leon Bottou Meta AI Research, FAIR NYC, USA leonb@meta.com Yann Le Cun Meta AI Research, FAIR, NYU NYC, USA ylecun@meta.com |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | No | All the codebase used to train the various models and to generate the figures is in the supplementary files. The full codebase and all the saved models will be released upon completion of the review process. The full code will be released with a NC by NC license on Git Hub. The full codebase along with pre-trained models will be released upon completion of the review process. |
| Open Datasets | Yes | a resnet50 trained on Imagenet sees its barn spider test accuracy falls from 68% to 46% only by introducing random crop DA during training. The inaturalist species classification and detection dataset [Van Horn et al., 2018]. Imagenet: A large-scale hierarchical image database [Deng et al., 2009]. |
| Dataset Splits | Yes | splitting the dataset into train/valid/test sets to optimizing the system s parameters e.g. from gradient updates of the loss on the train set while cross-validating hyper-parameters using the valid set, and (iii) assessing the system s performance on the test set. Cross-validation commonly involves many different types of regularization along with their strengths. |
| Hardware Specification | Yes | All the experiments ran on the Meta AI Research internal cluster equipped with NVIDIA A100 GPUs and Py Torch. |
| Software Dependencies | No | The paper mentions 'Py Torch' but does not specify a version number or list other software dependencies with versions. |
| Experiment Setup | Yes | Training settings. All models were trained for 100 epochs using SGD with momentum (0.9), batch size of 256 and initial learning rate of 0.1, decayed by a factor of 10 at epochs 30, 60, and 90. Weight decay and dropout were set to 0.0001 and 0.5 respectively when applicable. official Py Torch resnet50 implementation trained on Imagenet with horizontal flip and varying random crop lower bound DA. |