Controllable Dynamic Appearance for Neural 3D Portraits

Details

• Personal Author:
  Athar S ; Bi S ; Luan F ; Samaras D ; Shu Z ; Sunkavavalli K ; Xu Z
• Description:
  Recent advances in Neural Radiance Fields (NeRFs) have made it possible to reconstruct and reanimate dynamic portrait scenes with control over head-pose, facial expressions and viewing direction. However, training such models assumes photometric consistency over the deformed region e.g. the face must be evenly lit as it deforms with changing head-pose and facial expression. Such photometric consistency across frames of a video is hard to maintain, even in studio environments, thus making the created reanimatable neural portraits prone to artefacts during reanimation. In this work, we propose CoDyNeRF, a system that enables the creation of fully controllable 3D portraits in real-world capture conditions. CoDyNeRF learns to approximate illumination dependent effects via a dynamic appearance model in the canonical space that is conditioned on predicted surface normals and the facial expressions and head-pose deformations. The surface normals prediction is guided using 3DMM normals that act as a coarse prior for the normals of the human head, where direct prediction of normals is hard due to rigid and non-rigid deformations induced by head-pose and facial expression changes. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls, and realistic lighting effects. [Description provided by NIOSH]
• Subjects:
  Occupational Health Safety
• Keywords:
  3D Photographic Equipment Photometry Three Dimensional Virtual Reality Visual Images
• ISBN:
  9798350362459
• Publisher:
  Institute of Electrical and Electronics Engineers (IEEE)
• Document Type:
  Text
• Funding:
  Cooperative Agreement
• Genre:
  Proceedings
• Place as Subject:
  California ; New York ; OSHA Region 2 ; OSHA Region 3 ; OSHA Region 9 ; Pennsylvania
• CIO:
  National Institute for Occupational Safety and Health (NIOSH)
• Topic:
  Workplace Safety & Health
• Location:
  North America
• NIOSHTIC Number:
  nn:20069870
• Citation:
  2024 International Conference on 3D Vision (3DV 2024), March 18-21, 2024, Davos, Switzerland. ; New York: Institute of Electrical and Electronics Engineers (IEEE), 2024 Mar; :697-706
• Federal Fiscal Year:
  2024
• Performing Organization:
  State University of New York, Stony Brook
• Peer Reviewed:
  False
• Start Date:
  20220701
• Source Full Name:
  2024 International Conference on 3D Vision (3DV 2024), March 18-21, 2024, Davos, Switzerland
• End Date:
  20260630
• Collection(s):
  National Institute for Occupational Safety and Health
• Main Document Checksum:
  urn:sha-512:cc5e6110778f90584ee65a3ac49056b389d5f37a6b4906fbd05fae3690e0d94cb20618806a004b0d51f803fc17b85d9d2f7a9b61121929869a95f82a01eddea2
• Download URL:
  https://stacks.cdc.gov/view/cdc/208210/cdc_208210_DS1.pdf
• File Type:
  [PDF - 1.71 MB ]