Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
On the Edge of Memorization in Diffusion Models
Authors: Sam Buchanan, Druv Pai, Yi Ma, Valentin De Bortoli
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Within this setting, we hypothesize that the memorization or generalization behavior of an underparameterized trained model is determined by the difference in training loss between an associated memorizing model and a generalizing model. To probe this hypothesis, we theoretically characterize a crossover point wherein the weighted training loss of a fully generalizing model becomes greater than that of an underparameterized memorizing model at a critical value of model (under)parameterization. We then demonstrate via carefully-designed experiments that the location of this crossover predicts a phase transition in diffusion models trained via gradient descent, validating our hypothesis. Ultimately, our theory enables us to analytically predict the model size at which memorization becomes predominant. Our work provides an analytically tractable and practically meaningful setting for future theoretical and empirical investigations. Code for our experiments is available at https://github.com/Druv Pai/diffusion_mem_gen. |
| Researcher Affiliation | Collaboration | Sam Buchanan TTIC Druv Pai UC Berkeley Yi Ma UC Berkeley, HKU Valentin De Bortoli Google Deep Mind |
| Pseudocode | No | The paper includes theoretical formulations and mathematical proofs, particularly in the Appendix, but does not present any explicitly labeled 'Pseudocode' or 'Algorithm' blocks with structured steps. |
| Open Source Code | Yes | Code for our experiments is available at https://github.com/Druv Pai/diffusion_mem_gen. |
| Open Datasets | Yes | In our experiments in Section 4.2, we generate synthetic colored Fashion MNIST data by sampling K Fashion MNIST [Xiao et al., 2017] images uniformly at random to use as templates |
| Dataset Splits | No | The paper describes generating synthetic data and using a "held-out set of samples from π" for evaluation, but does not provide specific percentages, sample counts, or citations for predefined train/test/validation splits of a fixed dataset. |
| Hardware Specification | Yes | We run all experiments on several Nvidia A100 80GB GPUs using Jax 0.6.0 and Equinox 0.12 [Kidger and Garcia, 2021]. Each training/evaluation job occurs on a single A100 and the results are saved to file to be aggregated later. |
| Software Dependencies | Yes | We run all experiments on several Nvidia A100 80GB GPUs using Jax 0.6.0 and Equinox 0.12 [Kidger and Garcia, 2021]. |
| Experiment Setup | Yes | For all experiments, we use the variance preserving process which yields αt = 1 t2 and σt = t for t [0, 1]. We use the objective (5) to train our model denoisers. For training, we always train with the loss weighting λ(t) := α2 t/σ2 t , which is equivalent to using noise prediction ([Karras et al., 2022]), and t Unif((tℓ)L ℓ=0) where we use L = 25 decreasing timesteps tℓ= 0.01+0.998(L ℓ)/L = 0.999 0.998ℓ/L (0, 1). In lieu of computing the (obviously intractable) inner expectation in LN,t, we use Ndup := 100 Gaussian noise draws for each of the N samples to estimate the expectation. We use full-batch Adam for Nepochs epochs (also, iterations) to optimize the objective; for experiments in Section 4.1 we have Nepochs = 50, 000 and for experiments in Section 4.2 we have Nepochs = 100, 000. We use a warmup-decay learning rate schedule: for Nwarmup := Nepochs/10 epochs the learning rate linearly increases from 0 to 10 3; for the remaining Ndecay := Nepochs Nwarmup epochs the learning rate linearly decreases from 10 3 to 10 6. |