Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models
Authors: Giannis Daras, Weili Nie, Karsten Kreis, Alex Dimakis, Morteza Mardani, Nikola Kovachki, Arash Vahdat
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the effectiveness of our method for video inpainting and 8 video super-resolution, outperforming existing techniques based on noise transformations. We provide generated video results in the following URL: https://giannisdaras.github.io/warped_diffusion.github.io/. We extensively validate our method on video inpainting and super-resolution. |
| Researcher Affiliation | Collaboration | Giannis Daras UT Austin Weili Nie NVIDIA Karsten Kreis NVIDIA Alexandros G. Dimakis UT Austin Morteza Mardani NVIDIA Nikola B. Kovachki NVIDIA Arash Vahdat NVIDIA |
| Pseudocode | Yes | Algorithm 1 Warped Diffusion Temporal Consistency with Equivariance Self Guidance |
| Open Source Code | No | We do not yet release our code for noise warping and sampling guidance, but we are working on open-sourcing it. |
| Open Datasets | Yes | We train models with and without correlated noise on the COYO dataset [12] for 100k steps. |
| Dataset Splits | No | The paper mentions a 'test split of the COYO dataset' but does not specify a distinct validation set split (e.g., percentages or counts for training, validation, and test splits). |
| Hardware Specification | Yes | We train all our models on 16 A100 GPUs on a SLURM-based cluster. We perform all our sampling experiments on a single A-100 GPU. |
| Software Dependencies | No | The paper mentions software like Stable Diffusion XL, RAFT model, and PyTorch, but does not provide specific version numbers for these software components or programming languages (e.g., 'PyTorch 1.9', 'Python 3.8'). |
| Experiment Setup | Yes | We use the following training hyperparameters: Training resolution: 1024 1024. Batch size: 64. Latent resolution: 128. Optimizer Adam with Weight Decay. Optimizer parameters: Learning rate: 5e 6 β1 = 0.9 β2 = 0.999 Weight Decay: 1e 2 ϵ = 1e 08 Max Gradient Norm (Gradient clipping): 1.0 |