Ambient Diffusion: Learning Clean Distributions from Corrupted Data
Authors: Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, Adam Klivans
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We train models on standard benchmarks (Celeb A, CIFAR-10 and AFHQ) and show that we can learn the distribution even when all the training samples have 90% of their pixels missing. We also show that we can finetune foundation models on small corrupted datasets (e.g. MRI scans with block corruptions) and learn the clean distribution without memorizing the training set. 5 Experimental Evaluation |
| Researcher Affiliation | Collaboration | Giannis Daras UT Austin giannisdaras@utexas.edu Kulin Shah UT Austin kulinshah@utexas.edu Yuval Dagan UC Berkeley yuvald@berkeley.edu Aravind Gollakota Apple aravindg@cs.utexas.edu Alexandros G. Dimakis UT Austin dimakis@austin.utexas.edu Adam Klivans UT Austin klivans@utexas.edu |
| Pseudocode | Yes | We include pseudocode for our sampling in Algorithm 1. In the Appendix (Section E.4), we ablate an alternative choice for sampling that can lead to slight improvements at the cost of increased function evaluations. Algorithm 1 Ambient Diffusion Fixed Mask Sampling (for VE SDE) |
| Open Source Code | Yes | We open-source our code and models: https://github.com/giannisdaras/ambient-diffusion. |
| Open Datasets | Yes | We train models on standard benchmarks (Celeb A, CIFAR-10 and AFHQ) and Huggingface Brain Tumor MRI Dataset . In: (2020). URL: https://huggingface.co/datasets/miladfa7/Brain-MRI-Images-for-Brain-Tumor-Detection/ |
| Dataset Splits | No | The paper mentions training on standard benchmarks (Celeb A, CIFAR-10, AFHQ) and using subsets for finetuning (e.g., 3000 images from Celeb A), but it does not specify explicit train/validation/test split percentages, sample counts, or refer to specific predefined split files for reproducibility. |
| Hardware Specification | Yes | Our CIFAR-10 models required 2 days of training each on 6 A100 GPUs. Our AFHQ and Celeb A-HQ models required 6 days of training each on 6 A100 GPUs. Training for 15000 iterations takes 10 hours on an A100 GPU. |
| Software Dependencies | No | We use the EDM [30] codebase to train our models. We access Deepfloyd s IF [2] model through the diffusers library. The paper mentions specific implementations and libraries but does not provide explicit version numbers for them or other key software components like PyTorch or Python. |
| Experiment Setup | Yes | We list the rest of the hyperparameters we used in Table 2. Table 2: Training Hyperparameters Dataset Iters Batch LR SDE p δ Aug. Prob. Ch. Multipliers Dropout. For the finetuning, we set the training batch size to 32 and the learning rate to 3e-6. |