Carnegie Mellon University

Gm Global Technolgoy Operations, Inc.

Meta Platforms Technologies, LLC

Facebook, Inc.

Disney Enterprises, Inc.

Meta Platforms, Inc.

Fernando De La Torre

Dong Huang

Wende Zhang

Dan Levi

Yaser Sheikh

Debbie E Nachtegall

Xuehan Xiong

Michael Zollhoefer

Alexander Richard

Francisco Vicente Carrasco

Iain Matthews

Jason Saragih

Tomas Simon Kreuz

Jeffrey F Cohn

Jose Rafael Tena

Lele Chen

Vasu Agrawal

Chen Cao

Francisco Vicente

Zehua Huang

Franscisco Vicente

A method for training a three-dimensional model face animation model from speech, is provided. The method includes determining a first correlation value for a facial feature based on an audio waveform from a first subject, generating a first mesh for a lower portion of a human face, based on the facial feature and the first correlation value, updating the first correlation value when a difference between the first mesh and a ground truth image of the first subject is greater than a pre-selected threshold, and providing a three-dimensional model of the human face animated by speech to an immersive reality application accessed by a client device based on the difference between the first mesh and the ground truth image of the first subject. A non-transitory, computer-readable medium storing instructions to cause a system to perform the above method, and the system, are also provided.

Three-dimensional face animation from speech

A method for providing real-time three-dimensional facial animation from video is provided. The method includes collecting images of a subject, and forming a three-dimensional mesh for the subject based on a facial expression factor and a head pose of the subject extracted from the images of the subject. The method also includes forming a texture transformation based on an illumination parameter associated with an illumination configuration for the images from the subject, forming a three-dimensional model for the subject based on the three-dimensional mesh and the texture transformation, determining a loss factor based on selected points in a test image from the subject and a rendition of the test image by the three-dimensional model, and updating the three-dimensional model according to the loss factor. A system and a non-transitory, computer-readable medium storing instructions to perform the above method are also provided.

Real-time 3D facial animation from binocular video

Systems, methods, and non-transitory computer readable media can obtain a first image depicting a face of a user. A plurality of images depicting the face of the user can be identified. A second image of the plurality of images can be identified based on one or more factors. The face or a portion of the face of the user in the first image can be replaced with the face or a portion of the face of the user in the second image.

Systems and methods for swapping faces and face components based on facial recognition

Systems, methods, and non-transitory computer readable media can obtain a first image of a first user depicting a face of the first user with a neutral expression or position. A first image of a second user depicting a face of the second user with a neutral expression or position can be identified, wherein the face of the second user is similar to the face of the first user based on satisfaction of a threshold value. A second image of the first user depicting the face of the first user with an expression different from the neutral expression or position can be generated based on a second image of the second user depicting the face of the second user with an expression or position different from the neutral expression or position.

Systems and methods for generating personalized emoticons and lip synching videos based on facial recognition

The present invention relates to a system for detecting and tracking facial features in images and can be used in conjunction with a camera. Given a camera, the system will detect facial landmarks in images. The present invention includes software for real time, accurate facial feature detection and tracking in unconstrained images and videos. The present invention is better, more robust and faster than existing approaches and can be implemented very efficiently allowing real-time processing, even on low-power devices, such as mobile phones.

System and method for detecting and tracking facial features in images

A system and method for real-time image and video face de-identification that removes the identity of the subject while preserving the facial behavior is described The facial features of the source face are replaced with that of the target face while preserving the facial actions of the source face on the target face. The facial actions of the source face are transferred to the target face using personalized Facial Action Transfer (FAT), and the color and illumination is adapted. Finally, the source image or video containing the target facial features is outputted for display. Alternatively, the system can run in real-time.

System and method for processing video to provide facial de-identification

A method for monitoring a vehicle operator can be executed by a controller and includes the following steps: (a) receiving image data of a vehicle operator's head; (b) tracking facial feature points of the vehicle operator based on the image data; (c) creating a 3D model of the vehicle operator's head based on the facial feature points in order to determine a 3D position of the vehicle operator's head; (d) determining a gaze direction of the vehicle operator based on a position of the facial feature points and the 3D model of the vehicle operator's head; (e) determining a gaze vector based on the gaze direction and the 3D position of the vehicle operator's head; and (f) commanding an indicator to activate when the gaze vector is outside a predetermined parameter.

Vehicle operator monitoring system and method

A method for detecting an eyes-off-the-road condition based on an estimated gaze direction of a driver of a vehicle includes monitoring facial feature points of the driver within image input data captured by an in-vehicle camera device. A location for each of a plurality of eye features for an eyeball of the driver is detected based on the monitored facial features. A head pose of the driver is estimated based on the monitored facial feature points. The gaze direction of the driver is estimated based on the detected location for each of the plurality of eye features and the estimated head pose.

Driver gaze detection system

A method for determining an Eyes-Off-The-Road (EOTR) condition exists includes capturing image data corresponding to a driver from a monocular camera device. A detection of whether the driver is wearing eye glasses based on the image data using an eye glasses classifier. When it is detected that the driver is wearing eye glasses, a driver face location is detected from the captured image data and it is determined whether the EOTR condition exists based on the driver face location using an EOTR classifier.

Eyes-off-the-road classification with glasses classifier

An improved modeling system and associated techniques are described herein. In various embodiments, a modeling system generates a spatially local PCA model where the parts are connected with continuity constraints (e.g., soft constraints) in the boundaries. Experimental results on 3D face modeling show that the spatially local PCA model generalizes better than a holistic model. Moreover, the modeling system smoothly varies local control points for face posing in animation.

Growing community of inventors

Pittsburgh, PA, United States of America

Fernando De La Torre