Disney Enterprises, Inc.

Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Carnegie Mellon University

Denso Corporation

Toyota Motor Engineering & Manufacturing North America, Inc.

Yaser Sheikh

Jason Saragih

Iain Matthews

Tomas Simon Kreuz

Stephen Anthony Lombardi

Shih-En Wei

Mohsen Shahmohammadi

Hernan Badino

G Peter K Carr

Takaaki Shiratori

Fernando De La Torre

Michael Zollhoefer

Dawei Wang

Alexander Trenor Hypes

Michal Perdoch

Eric Foote

Patrick Joseph Lucey

Jessica Kate Hodgins

Sai Bi

Chenglei Wu

Shunsuke Saito

Shoou-I Yu

Kevyn Alex Anthony McPhail

Zhe Cao

Alexander Richard

Tomas Simon

Leonid Sigal

Takeo Kanade

Suhas E Chelian

Yukimasa Tamatsu

Hiroshi Sakai

George Peter Carr

James White

Ijaz Akhter

Alina Bialkowski

G Peter Carr

Ying Yang

Gabriel Bailowitz Schwartz

Paulo Urnau Gotardo

Natasha Kholgade

Sohaib Khan

Eakta Jain

Hyun Soo Park

Ankur Datta

Lele Chen

Vasu Agrawal

Qi Zhu

Chen Cao

Michael Peter K Mistry

Natasha Banerjee

A method for providing a relightable avatar of a subject to a virtual reality application is provided. The method includes retrieving multiple images including multiple views of a subject and generating an expression-dependent texture map and a view-dependent texture map for the subject, based on the images. The method also includes generating, based on the expression-dependent texture map and the view-dependent texture map, a view of the subject illuminated by a light source selected from an environment in an immersive reality application, and providing the view of the subject to an immersive reality application running in a client device. A non-transitory, computer-readable medium storing instructions and a system that executes the instructions to perform the above method are also provided.

Deep relightable appearance models for animatable face avatars

A method for training a three-dimensional model face animation model from speech, is provided. The method includes determining a first correlation value for a facial feature based on an audio waveform from a first subject, generating a first mesh for a lower portion of a human face, based on the facial feature and the first correlation value, updating the first correlation value when a difference between the first mesh and a ground truth image of the first subject is greater than a pre-selected threshold, and providing a three-dimensional model of the human face animated by speech to an immersive reality application accessed by a client device based on the difference between the first mesh and the ground truth image of the first subject. A non-transitory, computer-readable medium storing instructions to cause a system to perform the above method, and the system, are also provided.

Three-dimensional face animation from speech

A method for training a real-time, modeling for animating an avatar for a subject is provided. The method includes collecting multiple images of a subject. The method also includes selecting a plurality of vertex positions in a guide mesh, indicative of a volumetric primitive enveloping the subject, determining a geometric attribute for the volumetric primitive including a position, a rotation, and a scale factor of the volumetric primitive, determining a payload attribute for each of the volumetric primitive, the payload attribute including a color value and an opacity value for each voxel in a voxel grid defining the volumetric primitive, determining a loss factor for each point in the volumetric primitive based on the geometric attribute, the payload attribute and a ground truth value, and updating a three-dimensional model for the subject. A system and a non-transitory, computer-readable medium storing instructions to perform the above method are also provided.

Mixture of volumetric primitives for efficient neural rendering

A method for providing real-time three-dimensional facial animation from video is provided. The method includes collecting images of a subject, and forming a three-dimensional mesh for the subject based on a facial expression factor and a head pose of the subject extracted from the images of the subject. The method also includes forming a texture transformation based on an illumination parameter associated with an illumination configuration for the images from the subject, forming a three-dimensional model for the subject based on the three-dimensional mesh and the texture transformation, determining a loss factor based on selected points in a test image from the subject and a rendition of the test image by the three-dimensional model, and updating the three-dimensional model according to the loss factor. A system and a non-transitory, computer-readable medium storing instructions to perform the above method are also provided.

Real-time 3D facial animation from binocular video

The disclosed computer system may include an input module, an autoencoder, and a rendering module. The input module may receive geometry information and images of a subject. The geometry information may be indicative of variation in geometry of the subject over time. Each image may be associated with a respective viewpoint and may include a view-dependent texture map of the subject. The autoencoder may jointly encode texture information and the geometry information to provide a latent vector. The autoencoder may infer, using the latent vector, an inferred geometry and an inferred view-dependent texture of the subject for a predicted viewpoint. The rendering module may be configured to render a reconstructed image of the subject for the predicted viewpoint using the inferred geometry and the inferred view-dependent texture. Various other systems and methods are also disclosed.

Systems and methods for rendering avatars with deep appearance models

A virtual reality (VR) head mounted display (HMD) includes a light field camera on an outside surface of the HMD facing away from the wearer. Light rays and intensity captured by the light field camera is communicated to a console that identifies a virtual position of the light field camera based on a relative position of the light field camera to the user's eye when wearing the HMD. Based on the virtual position, the console selects rays of light captured by the light field camera projected to intersect a field of view of the light field camera if located at the virtual position. From the selected rays and corresponding intensities, the console generates one or more images representing the environment surrounding the HMD, providing a representation of portions of the environment surrounding the HMD visible from the position of the user's eye.

Generating content for presentation by a head mounted display based on data captured by a light field camera positioned on the head mounted display

An interactive system may include (1) a facial coupling subsystem configured to conduct a biopotential signal generated by a user's body, (2) a receiving subsystem electrically connected to the facial coupling subsystem and configured to receive, from the user's body via a compliant electrode of the facial coupling subsystem, the biopotential signal, and (3) a detection subsystem electrically connected to the receiving subsystem and configured to (a) determine a characteristic of the biopotential signal and (b) use the characteristic of the biopotential signal to determine a gaze direction of an eye of the user and/or a facial gesture of the user. In some examples, the facial coupling subsystem may include a plurality of compliant electrodes that each are configured to comply in a direction normal to a surface of the user's face. Various other apparatus, systems, and methods are also disclosed.

Apparatus, systems, and methods for sensing biopotential signals via compliant electrodes

A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes various image capture devices that capture images of portions of the user's face and body. Through image analysis, points of each portion of the user's face and body are identified from the images and their movement is tracked. The identified points are mapped to a three dimensional model of a face and to a three dimensional model of a body. From the identified points, animation parameters describing positioning of various points of the user's face and body are determined for each captured image. From the animation parameters and transforms mapping the captured images to three dimensions, the three dimensional model of the face and the three dimensional model of the body is altered to render movement of the user's face and body.

Generating graphical representation of a user's face and body using a monitoring system included on a head mounted display

Systems and methods for articulated pose estimation are provided. Some embodiments include training a convolutional neural network for object pose estimation, which includes receiving a two-dimensional training image of an articulated object that has a plurality of components and identifying, from the two-dimensional training image, at least one key point for each of the plurality of components. Some embodiments also include testing the accuracy of the object pose estimation, which includes visualizing a three or more dimensional pose of each of the plurality of components of the articulated object from a two-dimensional testing image and providing data related to the visualization for output.

Systems and methods for articulated pose estimation

A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes various image capture devices that capture images of portions of the user's face. Through image analysis, points of each portion of the user's face are identified from the images and their movement is tracked. The identified points are mapped to a three dimensional model of a face. From the identified points, a blendshape vector is determined for each captured image, resulting in various vectors indicating the user's facial expressions. A direct expression model that directly maps images to blendshape coefficients for a set of facial expressions based on captured information from a set of users may augment the blendshape vector in various embodiments. From the blendshape vectors and transforms mapping the captured images to three dimensions, the three dimensional model of the face is altered to render the user's facial expressions.

Generating graphical representation of facial expressions of a user wearing a head mounted display accounting for previously captured images of the user's facial expressions

A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes multiple image capture devices positioned within and on the HMD to capture portions of a face of a user wearing the HMD. Multiple image capture devices are included within the HMD to capture different portions of the face of the user within the HMD, and one or more other image capture devices are positioned to capture portions of the face of the user external to the HMD. Captured images from various image capture devices may be communicated to a console or a controller that generates a graphical representation of the user's face based on the captured images.

Positioning cameras in a head mounted display to capture images of portions of a face of a user

A system and method generates a broadcast image. The method includes receiving a first image captured by a first camera having a first configuration incorporating a first center of projection. The method includes determining a first mapping from a first image plane of the first camera onto a sphere. The method includes determining a second mapping from the sphere to a second image plane of a second virtual camera having a second configuration incorporating the first center of projection of the first configuration. The method includes generating a second image based upon the first image and a concatenation of the first and second mappings.

Method and device for hybrid robotic/virtual pan-tilt-zoom cameras for autonomous event recording

Growing community of inventors

Pittsburgh, PA, United States of America

Yaser Sheikh