DreamWaltz: Make a Scene with Complex 3D Animatable Avatars
Authors: Yukun Huang, Jianan Wang, Ailing Zeng, He CAO, Xianbiao Qi, Yukai Shi, Zheng-Jun Zha, Lei Zhang
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive evaluations demonstrate that Dream Waltz is an effective and robust approach for creating 3D avatars that can take on complex shapes and appearances as well as novel poses for animation. The proposed framework further enables the creation of complex scenes with diverse compositions, including avatar-avatar, avatar-object and avatar-scene interactions. ... We validate the effectiveness of our proposed framework for avatar generation and animation. In Sec. 4.1, we evaluate avatar generation with extensive text prompts for both qualitative comparisons and user studies. In Sec. 4.2, we demonstrate avatar animation given novel motion sequences. We present ablation analysis in Sec. 4.3 and illustrate that our framework can be further applied to make complex scenes with diverse interactions in Sec. 4.4. |
| Researcher Affiliation | Collaboration | Yukun Huang1,2 , Jianan Wang1 , Ailing Zeng1, He Cao1, Xianbiao Qi1, Yukai Shi1, Zheng-Jun Zha2, Lei Zhang1 1International Digital Economy Academy (IDEA) 2University of Science and Technology of China |
| Pseudocode | No | The paper describes its methods in text and with diagrams but does not include any pseudocode or algorithm blocks. |
| Open Source Code | No | The paper provides a link to a project page: 'See https://dreamwaltz3d.github.io/ for more vivid 3D avatar and animation results.' and 'Please refer to the project page at https://dreamwaltz3d.github.io/ for more animation sequences.' This link is for viewing results and demos, not for accessing the source code for the methodology. |
| Open Datasets | Yes | To create animation demonstrations, we utilize SMPL-format motion sequences from the 3DPW [42] and AIST++ [17] datasets to animate avatars. |
| Dataset Splits | No | The paper mentions data usage and sampling strategies (e.g., 'randomly sample the timestep from a uniform distribution', 'randomly sample SMPL pose parameters'), but it does not specify explicit training, validation, and test dataset splits with percentages or counts. |
| Hardware Specification | Yes | Dream Waltz is implemented in Py Torch and can be trained and evaluated on a single NVIDIA 3090 GPU. |
| Software Dependencies | No | The paper mentions 'Dream Waltz is implemented in Py Torch' and 'We adopt Control Net [48] with Stable-Diffusion v1.5 [34] as the backbone', but it does not specify version numbers for PyTorch or other key software components. |
| Experiment Setup | Yes | For the canonical avatar creation stage, we train the avatar representation for 30,000 iterations, which takes about an hour. For the animatable avatar learning stage, the avatar representation and the introduced density weighting network are further trained for 50,000 iterations. ... During training, we randomly sample the timestep from a uniform distribution of [20, 980], and the classifier-free guidance scale is set to 50.0. The weight term w(t) of SDS loss is set to 1.0, and we normalize the SDS gradients to stabilize the optimization process. The conditioning scale for Control Net is set to 1.0 by default. ... Throughout the entire training process, we use Adam [16] optimizer with a learning rate of 1e-3, and batch size is set to 1. |