Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
Authors: Alessandro Favero, Francesco Cagnetta, Matthieu Wyart
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We confirm our predictions on β empirically. We conclude by proving, under a natural universality assumption, that performing kernel regression with a ridge that decreases with the size of the training set leads to similar learning curve exponents to those we obtain in the ridgeless case. We confirm systematically our predictions by performing kernel ridgeless regression numerically for various t, s and embedding dimension d. Figure 1: Learning curves for different combinations of convolutional teachers with convolutional (left panels) and local (right panels) students. |
| Researcher Affiliation | Academia | Alessandro Favero Institute of Physics École Polytechnique Fédérale de Lausanne alessandro.favero@epfl.ch Francesco Cagnetta Institute of Physics École Polytechnique Fédérale de Lausanne francesco.cagnetta@epfl.ch Matthieu Wyart Institute of Physics École Polytechnique Fédérale de Lausanne matthieu.wyart@epfl.ch |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not explicitly state that source code for the described methodology is available, nor does it provide a link to a code repository. |
| Open Datasets | Yes | Moreover, we report the learning curves of local kernels on the CIFAR-10 dataset |
| Dataset Splits | No | The paper discusses generalisation error and training samples (P), but does not explicitly provide details on how the dataset was split into training, validation, and test sets (e.g., specific percentages or sample counts for each split). |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used to run the experiments (e.g., CPU/GPU models, memory, or cloud computing instances). |
| Software Dependencies | No | The paper mentions the use of certain kernels (e.g., Laplacian constituent kernels) but does not list any specific software libraries, frameworks, or their version numbers that would be necessary for reproduction. |
| Experiment Setup | Yes | We consider different combinations of convolutional and local teachers and students with overlapping patches and Laplacian constituent kernels, i.e. C(xi xj) = e xi xj . In order to test the robustness of our results to the data distribution, data are uniformly generated in the hypercube [0, 1]d (results in Fig. 1) or on a d-hypersphere (results in Appendix G). Panels A and B show that, with αt = αs = 1, our prediction β = 1/s holds independently of the embedding dimension d. Furthermore, notice that fixing the dimension d and the teacher filter size t, the generalisation errors of a convolutional and a local student with the same filter size differ only by a multiplicative constant independent of P. Panels C and D show learning curves for several values of s and fixed t. Panels E and F show learning curves for fixed t and s being smaller than, equal to or larger than t. |