Meta Platforms Technologies, LLC

Meta Platforms, Inc.

Michael Zollhoefer

Yaser Sheikh

Jason Saragih

Tomas Simon Kreuz

Stephen Anthony Lombardi

Fernando De La Torre

Shunsuke Saito

Amit Raj

Changil Kim

Alexander Richard

James Henry Hays

Brian M Wheelwright

Nathan Seigo Matsuda

Steven John Lovegrove

Joel Hegland

Johannes Peter Kopf

Jia-Bin Huang

Simon Gareth Green

Tanner Schmidt

Tianye Li

Michael Goesele

Gabriel Bailowitz Schwartz

Zhaoyang Lv

Miroslava Slavcheva

Christoph Lassner

Benjamin Attal

In particular embodiments, a computing system may access a set of training images for training a neural light field network for view synthesis. Using each training image, the computing system may train the neural light field network by casting, for each pixel of the training image, a ray into a three-dimensional (3D) space, the ray including integrated radiance along the ray, mapping first ray coordinates of the ray into an embedding network, transforming, using the embedding network, the first ray coordinates into second ray coordinates, applying positional encoding to second ray coordinates, generating, using the neural light field network, a predicted color value for the pixel based on positionally encoded second ray coordinates, comparing the predicted color value for the pixel with a ground-truth color value for the pixel, and updating the neural light field network and the embedding network based on comparison.

Learning neural light fields with ray-space embedding networks

In one embodiment, a method includes initializing latent codes respectively associated with times associated with frames in a training video of a scene captured by a camera. For each of the frames, a system (1) generates rendered pixel values for a set of pixels in the frame by querying NeRF using the latent code associated with the frame, a camera viewpoint associated with the frame, and ray directions associated with the set of pixels, and (2) updates the latent code associated with the frame and the NeRF based on comparisons between the rendered pixel values and original pixel values for the set of pixels. Once trained, the system renders output frames for an output video of the scene, wherein each output frame is rendered by querying the updated NeRF using one of the updated latent codes corresponding to a desired time associated with the output frame.

Neural 3D video synthesis

A method for training a three-dimensional model face animation model from speech, is provided. The method includes determining a first correlation value for a facial feature based on an audio waveform from a first subject, generating a first mesh for a lower portion of a human face, based on the facial feature and the first correlation value, updating the first correlation value when a difference between the first mesh and a ground truth image of the first subject is greater than a pre-selected threshold, and providing a three-dimensional model of the human face animated by speech to an immersive reality application accessed by a client device based on the difference between the first mesh and the ground truth image of the first subject. A non-transitory, computer-readable medium storing instructions to cause a system to perform the above method, and the system, are also provided.

Three-dimensional face animation from speech

A method of forming a pixel-aligned volumetric avatar includes receiving multiple two-dimensional images having at least two or more fields of view of a subject. The method also includes extracting multiple image features from the two-dimensional images using a set of learnable weights, projecting the image features along a direction between a three-dimensional model of the subject and a selected observation point for a viewer, and providing, to the viewer, an image of the three-dimensional model of the subject. A system and a non-transitory, computer readable medium storing instructions to perform the above method, are also provided.

Pixel-aligned volumetric avatars

A device for providing a reverse pass-through view of a user of a headset display to an onlooker includes an eyepiece comprising an optical surface configured to provide an image to a user on a first side of the optical surface. The device also includes a first camera configured to collect an image of a portion of a face of the user reflected from the optical surface in a first field of view, a display adjacent to the optical surface and configured to project forward an image of the face of the user, and a screen configured to receive light from the display and provide the image of the face of the user to an onlooker.

Reverse pass-through glasses for augmented reality and virtual reality devices

A method for training a real-time, modeling for animating an avatar for a subject is provided. The method includes collecting multiple images of a subject. The method also includes selecting a plurality of vertex positions in a guide mesh, indicative of a volumetric primitive enveloping the subject, determining a geometric attribute for the volumetric primitive including a position, a rotation, and a scale factor of the volumetric primitive, determining a payload attribute for each of the volumetric primitive, the payload attribute including a color value and an opacity value for each voxel in a voxel grid defining the volumetric primitive, determining a loss factor for each point in the volumetric primitive based on the geometric attribute, the payload attribute and a ground truth value, and updating a three-dimensional model for the subject. A system and a non-transitory, computer-readable medium storing instructions to perform the above method are also provided.

Mixture of volumetric primitives for efficient neural rendering

Growing community of inventors

Pittsburgh, PA, United States of America

Michael Zollhoefer