The current demand for 3D content for AR glasses, VR headsets, or 3D monitors motivates to create 3D models obtainable by a simple mobile device.
A recent study proposes a method for constructing a relightable 3D human head portrait. It uses a handheld video of a person made by a smartphone with a blinking flash.
Image credit: Christian Widell via Wikimedia
A camera pose is estimated for every frame, and a dense point cloud is reconstructed. The points and their descriptions are then rasterized for novel camera viewpoints by a novel rendering network. The rasterizations are translated into a series of maps, and the image is reconstructed accordingly to the parameters of the scene lighting and the camera flash.
Due to limited photometric information in the video, human face-specific priors are also used for the reconstruction. The recreated scene can be rendered from new viewpoints and with new lighting sources at real-time speeds.
Link to the page of project: https://saic-violet.github.io/relightable-portrait/