Spectral Bias in Practice: The Role of Function Frequency in Generalization
Authors: Sara Fridovich-Keil, Raphael Gontijo Lopes, Rebecca Roelofs
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this work, we propose methodologies for measuring spectral bias in modern image classification networks on CIFAR-10 and Image Net. We find that these networks indeed exhibit spectral bias, and that interventions that improve test accuracy on CIFAR-10 tend to produce learned functions that have higher frequencies overall but lower frequencies in the vicinity of examples from each class. |
| Researcher Affiliation | Collaboration | Sara Fridovich-Keil University of California, Berkeley sfk@eecs.berkeley.edu Raphael Gontijo-Lopes Google Brain iraphael@google.com Rebecca Roelofs Google Brain rofls@google.com |
| Pseudocode | No | The paper describes the methods in narrative text and mathematical formulations but does not include any explicit pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our CIFAR-10 code is available at https://github.com/google-research/google-research/tree/master/spectral_bias; for Image Net we apply the same interpolation method to pre-cropped images and pretrained model checkpoints. |
| Open Datasets | Yes | We use the CIFAR-10 [23] dataset of low-resolution (32 32) natural images from ten animal and object classes and the Image Net [8] dataset of higher-resolution (224 224) natural images from 1000 classes. |
| Dataset Splits | Yes | Our label smoothing experiments use the CIFAR-10 dataset [23], where ntrain = 50000, nval = 10000, d = 32, c = 3, and M = 10. |
| Hardware Specification | No | These results did require substantial GPU compute, but we do not quantify the exact amount. Each point in a label smoothing figure required training a model from scratch, and each value in an interpolation figure required evaluating a pretrained model on hundreds of thousands of images. |
| Software Dependencies | No | The paper does not explicitly list software dependencies with version numbers, such as Python or specific deep learning frameworks and libraries. |
| Experiment Setup | Yes | We train from scratch using the original examples Xi and their smoothed labels yi. We train a Wide Res Net32 model (wide-resnet with width 32) with radial wave label smoothing at frequency 0.04. ... training we tested. ... Mixup (with strength 0.1) ... weight decay and early stopping. |