Finding Generalization Measures by Contrasting Signal and Noise
Authors: Jiaye Teng, Bohang Zhang, Ruichen Li, Haowei He, Yequan Wang, Yan Tian, Yang Yuan
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, REF Complexity monotonically relates to test accuracy in real-world datasets without accessing additional validation sets, achieving 0.988 correlation on CIFAR-10 and 0.960 correlation on CIFAR-100. We further theoretically verify the utility of REF Complexity under three different cases... |
| Researcher Affiliation | Collaboration | 1Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China 2Peking University 3Beijing Academy of Artificial Intelligence, Beijing, China 4Shanghai Artificial Intelligence Laboratory 5Shanghai Qi Zhi Institute. |
| Pseudocode | Yes | We summarize the algorithm in Algorithm 1, which returns the REF Complexity value T β n (D, At). |
| Open Source Code | Yes | The code is available at https://github.com/962086838/REF-complexity. |
| Open Datasets | Yes | Specifically, we conduct over one hundred experiments on CIFAR-10 and CIFAR-100, and plot each regime s test accuracy and REF Complexity in Figure 1. |
| Dataset Splits | No | The paper states it achieves its results 'without accessing additional validation sets' and discusses why traditional validation sets are unnecessary. Therefore, it does not provide validation dataset splits. |
| Hardware Specification | Yes | We run all the experiments on a RTX2080Ti graphic card. |
| Software Dependencies | No | The paper mentions using SGD, but does not specify any software libraries or their version numbers (e.g., PyTorch, TensorFlow, Python versions) used in the experiments. |
| Experiment Setup | Yes | We train each model for 150 epochs. To evaluate REF Complexity correctly, each noise training process is trained five times... Hyperparameters: learning rate 0.1, 0.01, 0.001; batch size 256, 512, 1024; weight decay 1e-5, 1e-6, 1e-7; architecture Res Net20, Res Net32, Res Net44, Res Net56. |