Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Time-Independent Information-Theoretic Generalization Bounds for SGLD
Authors: Futoshi Futami, Masahiro Fujisawa
NeurIPS 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | We provide novel information-theoretic generalization bounds for stochastic gradient Langevin dynamics (SGLD) under the assumptions of smoothness and dissipativity, which are widely used in sampling and non-convex optimization studies. Our bounds are time-independent and decay to zero as the sample size increases, regardless of the number of iterations and whether the step size is fixed. Unlike previous studies, we derive the generalization error bounds by focusing on the time evolution of the Kullback Leibler divergence, which is related to the stability of datasets and is the upper bound of the mutual information between output parameters and an input dataset. Additionally, we establish the first information-theoretic generalization bound when the training and test loss are the same by showing that a loss function of SGLD is sub-exponential. This bound is also time-independent and removes the problematic step size dependence in existing work, leading to an improved excess risk bound by combining our analysis with the existing non-convex optimization error bounds. |
| Researcher Affiliation | Academia | Futoshi Futami Osaka University / RIKEN AIP EMAIL Masahiro Fujisawa RIKEN AIP EMAIL |
| Pseudocode | No | The paper is theoretical and focuses on mathematical derivations and proofs; it does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any concrete access to source code, such as a repository link or an explicit statement about code release. |
| Open Datasets | No | The paper is theoretical and does not use or provide concrete access information for a specific publicly available or open dataset for training purposes. |
| Dataset Splits | No | The paper is theoretical and does not conduct experiments, therefore it does not provide specific dataset split information for validation. |
| Hardware Specification | No | The paper is theoretical and does not describe any specific hardware used for running experiments. |
| Software Dependencies | No | The paper is theoretical and does not mention any specific ancillary software details with version numbers required for replication. |
| Experiment Setup | No | The paper is theoretical and does not describe any specific experimental setup details, such as hyperparameter values or training configurations. |