Improved Regularization and Robustness for Fine-tuning in Neural Networks
Authors: Dongyue Li, Hongyang Zhang
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We validate our approach on an extensive collection of image and text data sets using multiple pre-trained model architectures. Our approach improves baseline methods by 1.76% (on average) for seven image classification tasks and 0.75% for a few-shot classification task. When the target data set includes noisy labels, our approach outperforms baseline methods by 3.56% on average in two noisy settings. |
| Researcher Affiliation | Academia | Dongyue Li Northeastern University li.dongyu@northeastern.edu Hongyang R. Zhang Northeastern University ho.zhang@northeastern.edu |
| Pseudocode | Yes | Algorithm 1 Regularized self-labeling (REGSL) |
| Open Source Code | Yes | Our code is available at https://github.com/NEU-Stats ML-Research/Regularized-Self-Labeling. |
| Open Datasets | Yes | We consider seven image classification data sets described in Table 1. ... Aircrafts (Maji et al., 2013) CUB-200-2011 (Wah et al., 2011) Caltech-256 (Griffin et al., 2007) Stanford-Cars (Krause et al., 2013) Stanford-Dogs (Khosla et al., 2011) Flowers (Nilsback and Zisserman, 2008) MIT-Indoor (Sharif Razavian et al., 2014)... For the medical imaging task, we consider the Chest X-ray14 data set contains 112120 frontal-view chest X-ray images labeled with 14 different diseases (Wang et al., 2017; Rajpurkar et al., 2017). |
| Dataset Splits | Yes | Table 1: Basic statistics for seven image classification tasks. Datasets Training Validation Test Classes Aircrafts (Maji et al., 2013) 3334 3333 3333 100 CUB-200-2011 (Wah et al., 2011) 5395 599 5794 200 Caltech-256 (Griffin et al., 2007) 7680 5120 5120 256 Stanford-Cars (Krause et al., 2013) 7330 814 8441 196 Stanford-Dogs (Khosla et al., 2011) 10800 1200 8580 120 Flowers (Nilsback and Zisserman, 2008) 1020 1020 6149 102 MIT-Indoor (Sharif Razavian et al., 2014) 4824 536 1340 67 |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory, or cloud instance types) used for running the experiments. |
| Software Dependencies | No | The paper refers to certain software or frameworks (e.g., Optuna for hyperparameter optimization, Adam optimizer) but does not provide specific version numbers for these or other key software dependencies required for reproducibility. |
| Experiment Setup | Yes | We describe the fine-tuning procedure and the hyperparameters in Section B.1. We set four different values of Di for the four blocks of the Res Net models in our algorithm. |