Color and infra-red three-dimensional reconstruction using implicit radiance functions

Abstract

An image is rendered based a neural radiance field (NeRF) volumetric representation of a scene, where the NeRF representation is based on captured frames of video data, each frame including a color image, a widefield IR image, and a plurality of depth IR images of the scene. Each depth IR image is captured when the scene is illuminated by a different pattern of points of IR light, and the illumination by the patterns occurs at different times. The NeRF representation provides a mapping between positions and viewing directions to a color and optical density at each position in the scene, where the color and optical density at each position enables a viewing of the scene from a new perspective, and the NeRF representation provides a mapping between positions and viewing directions to IR values for each of the different patterns of points of IR light from the new perspective.