Google Inc.

Vivek Kwatra

Mei Han

Matthias Grundmann

Jay Yagnik

Michele Mae Covell

Saurabh Mathur

Irfan Essa

Shumeet Baluja

Sergey Ioffe

Shengyang Dai

Alexander Toshev

Ullas Gargi

Troy Chinen

Hui Fang

Mehmet Emre Sargin

Matthew Stephen Steiner

Erik Murphy-Chutorian

Nicholas Butko

Christian Frueh

Henry Hao Tang

Mayur Dhondu Datar

John Patrick Flynn

Allen Huang

Sangho Yoon

Florian Kriener

Avneesh Sud

Yichang Shih

Tom Binder

Steven Vandebogart

Nikolaos Trogkanis

Ingo Wehmeyer

Huei-Hung Christopher Liao

Avisek Lahiri

John Lewis

John Spiegel

Falk Sticken

Meng Zhang

Jingyu Cui

Dave Cohen

Hao Tang

Josh T Wills

Poonam Suryanarayan

Steven Hickson

Nicholas Dufour

Jing-yu Cui

Provided is a framework for generating photorealistic 3D talking faces conditioned only on audio input. In addition, the present disclosure provides associated methods to insert generated faces into existing videos or virtual environments. We decompose faces from video into a normalized space that decouples 3D geometry, head pose, and texture. This allows separating the prediction problem into regressions over the 3D face shape and the corresponding 2D texture atlas. To stabilize temporal dynamics, we propose an auto-regressive approach that conditions the model on its previous visual state. We also capture face illumination in our model using audio-independent 3D texture normalization.

Photorealistic talking faces from audio

A highlight learning technique is provided to detect and identify highlights in sports videos. A set of event models are calculated from low-level frame information of the sports videos to identify recurring events within the videos. The event models are used to characterize videos by detecting events within the videos and using the detected events to generate an event vector. The event vector is used to train a classifier to identify the videos as highlight or non-highlight.

Learning highlights using event detection

Images of a plurality of users are captured concurrently with the plurality of users evincing a plurality of expressions. The images are captured using one or more eye tracking sensors implemented in one or more head mounted devices (HMDs) worn by the plurality of first users. A machine learnt algorithm is trained to infer labels indicative of expressions of the users in the images. A live image of a user is captured using an eye tracking sensor implemented in an HMD worn by the user. A label of an expression evinced by the user in the live image is inferred using the machine learnt algorithm that has been trained to predict labels indicative of expressions. The images of the users and the live image can be personalized by combining the images with personalization images of the users evincing a subset of the expressions.

Classifying facial expressions using eye-tracking cameras

A computer-implemented method, computer program product, and computing system is provided for interacting with images having similar content. In an embodiment, a method may include identifying a plurality of photographs as including a common characteristic. The method may also include generating a flipbook media item including the plurality of photographs. The method may further include associating one or more interactive control features with the flipbook media item.

System and method for grouping related photographs

A camera captures an image of a user wearing a head mounted device (HMD) that occludes a portion of the user's face. A three-dimensional (3-D) pose that indicates an orientation and a location of the user's face in a camera coordinate system is determined. A representation of the occluded portion of the user's face is determined based on a 3-D model of the user's face. The representation replaces a portion of the HMD in the image based on the 3-D pose of the user's face in the camera coordinate system. In some cases, the 3-D model of the user's face is selected from 3-D models of the user's face stored in a database that is indexed by eye gaze direction. Mixed reality images can be generated by combining virtual reality images, unoccluded portions of the user's face, and representations of an occluded portion of the user's face.

Headset removal in virtual, augmented, and mixed reality using an eye gaze database

An example method is disclosed that includes identifying a training set of images, wherein each image in the training set has an identified bounding box that comprises an object class and an object location for an object in the image. The method also includes segmenting each image of the training set, wherein segments comprise sets of pixels that share visual characteristics, and wherein each segment is associated with an object class. The method further includes clustering the segments that are associated with the same object class, and generating a data structure based on the clustering, wherein entries in the data structure comprise visual characteristics for prototypical segments of objects having the object class and further comprise one or more potential bounding boxes for the objects, wherein the data structure is usable to predict bounding boxes of additional images that include an object having the object class.

System and method for using segmentation to identify object location in images

An easy-to-use online video stabilization system and methods for its use are described. Videos are stabilized after capture, and therefore the stabilization works on all forms of video footage including both legacy video and freshly captured video. In one implementation, the video stabilization system is fully automatic, requiring no input or parameter settings by the user other than the video itself. The video stabilization system uses a cascaded motion model to choose the correction that is applied to different frames of a video. In various implementations, the video stabilization system is capable of detecting and correcting high frequency jitter artifacts, low frequency shake artifacts, rolling shutter artifacts, significant foreground motion, poor lighting, scene cuts, and both long and short videos.

Cascaded camera motion estimation, rolling shutter detection, and camera shake detection for video stabilization

A method, computer program product, and computer system for identifying a first portion of a facial image in a first image, wherein the first portion includes noise. A corresponding portion of the facial image is identified in a second image, wherein the corresponding portion includes less noise than the first portion. One or more filter parameters of the first portion are determined based upon, at least in part, the first portion and the corresponding portion. At least a portion of the noise from the first portion is smoothed based upon, at least in part, the one or more filter parameters. At least a portion of face specific details from the corresponding portion is added to the first portion.

Image denoising system and method

Implementations generally relate to generating compositional media content. In some implementations, a method includes receiving a plurality of photos from a user, and determining one or more composition types from the photos. The method also includes generating compositions from the selected photos based on the one or more determined composition types. The method also includes providing the one or more generated compositions to the user.

Generating compositions

Methods and systems for processing a video for stabilization are described. A recorded video may be stabilized by removing at least a portion of shake introduced in the video. An original camera path for a camera used to record the video may be determined. A crop window size may be selected, a crop window transform may accordingly be determined, and the crop window transform may be applied to the original video to provide a modified video from a viewpoint of the modified motion camera path.

Methods and systems for processing a video for stabilization using dynamic crop

A computing device segments an image into a plurality of segments, wherein each segment of the plurality of segments has a segment location and a set of pixels that share visual characteristics. The computing device determines an initial set of bounding boxes for the image based on the plurality of segments. The computing device determines a reduced set of bounding boxes based on combining bounding boxes of the initial set of bounding boxes, the reduced set of bounding boxes corresponding to one or more objects in the image, each of the one or more objects having an object class and an object location.

Growing community of inventors

Santa Clara, CA, United States of America

Vivek Kwatra