FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models
Authors: Hao ZHANG, Tianyuan DAI, Yanbo Xu, Yu-Wing Tai, Chi-Keung Tang
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments show that our Face DNe RF achieves exceptionally realistic results and unprecedented flexibility in editing compared with state-of-the-art 3D face reconstruction and editing methods. This section presents our editing results and compare them with representative methods, emphasizing Face DNe RF s disentanglement capability which is conducive to editing control, its flexibility in utilizing a text-guided diffusion model, as well as the multi-view consistency in illumination control. Furthermore, our method can be migrated to other data domains with trained generative models. 4 Experiments 4.5 Ablation Study |
| Researcher Affiliation | Academia | Hao Zhang HKUST hzhangcc@connect.ust.hk Yanbo Xu CMU, HKUST yxubu@connect.ust.hk Tianyuan Dai Stanford University, HKUST tdaiaa@connect.ust.hk Yu-Wing Tai Dartmouth College yuwing@gmail.com Chi-Keung Tang HKUST cktang@cs.ust.hk |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our code will be available at https://github.com/Billy XYB/Face DNe RF. |
| Open Datasets | Yes | We utilize trained checkpoints of EG3D on FFHQ [25], AFHQv2 [11] and Shape Net [9] for the data domain of face, cat and car respectively. |
| Dataset Splits | No | We utilize trained checkpoints of EG3D on FFHQ [25], AFHQv2 [11] and Shape Net [9] for the data domain of face, cat and car respectively. The paper does not specify custom training, validation, or test splits for its own experiments, as it uses pre-trained models. |
| Hardware Specification | Yes | We set the optimization iterations for our editing to 500, which takes approximately 10 minutes on a 3090 GPU. |
| Software Dependencies | No | We replace the Py Torch in-place operations and other numpy operations in the Hn model with equivalent differentiable Py Torch tensor operations. The paper mentions PyTorch and numpy, but does not provide specific version numbers for these or other software dependencies. |
| Experiment Setup | Yes | We set the weighting LD, LID, LR to be 0.2, 0.2 and 2 10 5 for most editing cases, which can be finetuned for each editing. We set the optimization iterations for our editing to 500. The camera rotation angles θ and ϕ are randomly sampled from [ π 12] and [ π 12], where θ and ϕ are the angles of spherical coordinate. |