In search of robust measures of generalization
Authors: Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, Ioannis Mitliagkas, Daniel M. Roy
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We collect data from thousands of experiments on CIFAR-10 and SVHN, with various values for width, depth, learning rate, and training set size. |
| Researcher Affiliation | Collaboration | 1Element AI, 2Mila, 3Université de Montréal, 4University of Toronto, 5Vector Institute |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide an explicit statement about releasing its source code or a link to a repository for the methodology described. |
| Open Datasets | Yes | We collect data from thousands of experiments on CIFAR-10 and SVHN, with various values for width, depth, learning rate, and training set size. |
| Dataset Splits | No | The paper mentions 'training data' and 'held-out data' but does not explicitly provide specific train/validation/test dataset split percentages, sample counts, or a detailed splitting methodology for reproducibility. |
| Hardware Specification | No | The paper mentions 'massive computing resources' but does not provide specific hardware details such as GPU/CPU models, processor types, or memory specifications used for running experiments. |
| Software Dependencies | No | The paper refers to software such as PyTorch, NumPy, Matplotlib, and Scikit-learn in its references, but it does not specify version numbers for these or any other software dependencies needed to replicate the experiments. |
| Experiment Setup | Yes | In our experiments, variable assignments (!) are pairs (H, σ) of hyperparameter settings and random seeds, respectively. The hyperparameters are: learning rate, neural network width and depth; dataset (CIFAR-10 or SVHN), and training set size. (See Appendix C for ranges.) |