Regularizing neural radiance fields with denoising diffusion models

Abstract

Methods and systems for creating accurate three-dimensional representations of environments using neural radiance fields regularized by denoising diffusion models are disclosed. Receiving a plurality of images representing an environment, a scene representation model is trained to create the three-dimensional model of the environment. Using these images, the virtual rays are sampled from training viewpoints within the environment. The scene representation model is then applied to these rays to generate simulated images of the environment from the training viewpoints. These simulated images undergo a regularization process that uses a denoising diffusion model to determine color gradients and depth gradients in each simulated image. The scene representation model is trained with this data to create the final three-dimensional model of the environment. This model is provided to the requesting client device to generate the three-dimensional representation and create a virtual object within the environment.