Snap Inc.

Vladislav Shakhrai

Sergey Demyanov

Aleksei Stoliar

Mikhail Vasilkovskii

Istvan Marton

Ju Hu

Pavlo Chemerys

Junli Cao

A neural light field (NeLF) that runs real-time on mobile devices for neural rendering of three dimensional (3D) scenes, referred to as MobileR2L. The MobileR2L architecture runs efficiently on mobile devices with low latency and small size, and it achieves high-resolution generation while maintaining real-time inference for both synthetic and real-world 3D scenes on mobile devices. The MobileR2L has a network backbone including a convolutional layer embedding an input image at a resolution, residual blocks uploading the embedded image, and super-resolution modules receiving the uploaded embedded image and rendering an output image having a higher resolution than the embedded image. The convolution layer generates a number of rays equal to a number of pixels in the input image, where a partial number of the rays is uploaded to the super-resolution modules.

Real-time neural light field on mobile devices

A three-dimensional asset (3D) reconstruction technique for generating a 3D asset representing an object from images of the object. The images are captured from different viewpoints in a darkroom using one or more light sources having known locations. The system estimates camera poses for each of the captured images and then constructs a 3D surface mesh made up of surfaces using the captured images and their respective estimated camera poses. Texture properties for each of the surfaces of the 3D surface mesh are then refined to generate the 3D asset.

Three-dimensional asset reconstruction

The subject technology receives an object mesh, information related to a viewpoint for rendering an image of an object having a reflective surface, and a set of maps. The subject technology generates a rasterized RGB (Red Green Blue) image based on the object mesh, the viewpoint, and the set of maps. The subject technology generates, using a neural network model, an output image of the object with the reflective surface based at least in part on the rasterized RGB image and the viewpoint. The subject technology provides for display the output image of the object with the reflective surface on a display of a computer client device.

Neural shading of reflective surfaces

Methods and systems are disclosed for performing operations for generating a photorealistic rendering of an object. The operations include: accessing a set of albedo textures and a machine learning model associated with a real-world object, the set of albedo textures and a machine learning model having been trained based on a plurality of viewpoints of the real-world object; obtaining a three-dimensional (3D) mesh of the real-world object; receiving input that selects a new viewpoint that differs from the plurality of viewpoints of the real-world object; and generating a photorealistic rendering of the real-world object from the new viewpoint based on the 3D mesh of the real-world object, the set of albedo textures, and the machine learning model associated with the real-world object.

Growing community of inventors

London, United Kingdom

Vladislav Shakhrai