The Implicit Bias of Depth: How Incremental Learning Drives Generalization
Authors: Daniel Gissin, Shai Shalev-Shwartz, Amit Daniely
ICLR 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We complement our theoretical findings by experimenting with deep matrix sensing, quadratic neural networks and with binary classification using diagonal and convolutional linear networks, showing all of these models exhibit incremental learning. |
| Researcher Affiliation | Academia | School of Computer Science The Hebrew University Jerusalem, Israel |
| Pseudocode | No | The paper does not contain any pseudocode or algorithm blocks. |
| Open Source Code | Yes | The code for reproducing all of our experiments can be found in https://github.com/dsgissin/Incremental-Learning |
| Open Datasets | No | The paper refers to training on 'a small training set' and uses synthetic data (e.g., 'Gaussian inputs') but does not specify or provide access to any named publicly available datasets. |
| Dataset Splits | No | The paper mentions 'a small training set' but does not provide specific details on dataset splits (e.g., percentages, sample counts for train/validation/test sets), nor does it mention cross-validation. |
| Hardware Specification | No | The paper does not provide any specific hardware details (e.g., GPU/CPU models, memory, cloud instance types) used for running the experiments. |
| Software Dependencies | No | The paper mentions implementing models and using gradient descent but does not list specific software libraries or their version numbers (e.g., Python, PyTorch, TensorFlow versions). |
| Experiment Setup | Yes | All models were trained using gradient descent with a small initialization and learning rate, on a small training set such that there are multiple possible solutions. (Figure 1 caption). Gaussian initialization with variance such that the initial singular values are in expectation 1e-4. (Figure 4 caption). |