Do deep networks transfer invariances across classes?
Authors: Allan Zhou, Fahim Tajwar, Alexander Robey, Tom Knowles, George J. Pappas, Hamed Hassani, Chelsea Finn
ICLR 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through careful experimentation, we observe that invariance to class-agnostic transformations is still heavily dependent on class size, with the networks being much less invariant on smaller classes. This result holds even when using data balancing techniques, and suggests poor invariance transfer across classes. Our results provide one explanation for why classifiers generalize poorly on unbalanced and long-tailed distributions. |
| Researcher Affiliation | Academia | Allan Zhou & Fahim Tajwar Stanford University Alexander Robey University of Pennsylvania Tom Knowles Stanford University George J. Pappas & Hamed Hassani University of Pennsylvania Chelsea Finn Stanford University |
| Pseudocode | Yes | Algorithm 1 Generative Invariance Transfer: Classifier Training |
| Open Source Code | Yes | Source code for our experiments is available at https://github.com/Allan Yang Zhou/ generative-invariance-transfer. |
| Open Datasets | Yes | We modify Kuzushiji-49 (Clanuwat et al., 2018) to create three synthetic datasets using three different nuisance transformations: image rotation (K49-ROT-LT), varying background intensity (K49BG-LT), and image dilation or erosion (K49-DIL-LT). ... GTSRB-LT is a long-tailed variant of GTSRB (Stallkamp et al., 2012; 2011) ... CIFAR-10-LT and CIFAR-100-LT are long-tailed CIFAR (Krizhevsky, 2009) variants used in previous long-tailed classification literature (Cao et al., 2019; Tang et al., 2020). ... Tiny Image Net-LT is constructed from Tiny Image Net (Le & Yang, 2015) similarly to CIFAR-LT. ... i Naturalist is a large scale species detection dataset (Horn et al., 2018). |
| Dataset Splits | Yes | We resize every image to have dimensions 32x32, randomly sample 25% of our training set as the validation set and use the rest to construct the long-tailed training dataset. ... Since we are calculating class-balanced test metrics, we leave the test set unchanged. |
| Hardware Specification | No | The paper mentions training models and architectures (e.g., 'Res Net architecture', 'Efficient Net-b4 backbone') but does not provide specific hardware details such as GPU or CPU models used for the experiments. |
| Software Dependencies | No | The paper mentions software components like 'PyTorch', 'Adam', and 'SGD', but does not provide specific version numbers for these or other software dependencies. |
| Experiment Setup | Yes | For GTSRB-LT and CIFAR-LT we train Res Net32 (He et al., 2015) models for 200 epochs with batch size 128, optimized by SGD with momentum 0.9, weight decay 2 10 4, and initial learning rate 0.1. The learning rate decays by a factor of 10 at epochs 160 and 180. For K49-LT we use a Res Net20 backbone trained for 50 epochs, with learning rate decays at 30 and 40 epochs. |