Google Inc.

Apple Inc.

Faceshift Ag

Sofien Bouaziz

Mark Pauly

Shahram Izadi

Sean Ryan Francesco Fanello

Rohit Kumar Pandey

Thibaut Weise

Jonathan James Taylor

Cem Keskin

Yinda Zhang

Andrea Tagliasacchi

Christian Haene

Dongeek Shin

Paul E Debevec

Adarsh Prakash Murthy Kowdle

Christoph Rhemann

Hao Li

Ivan Poupyrev

Shwetak N Patel

Geoffrey Everest Hinton

Daniel Goldman

Seth Taylor Raphael

Philip L Davidson

David Kim

Ricardo Martin Brualla

Chloe LeGendre

Sameh Khamis

Danhang Tang

Brian Amberg

Mingsong Dou

Kyle Genova

Stefan Brugger

Reinhard Knothe

Thomas Kiser

Patrick Snape

Alex Mansfield

Boyang Deng

Soroosh Yazdani

Onur Gonen Guleryuz

Jonathan Tilton Barron

Ke Li

Jason Spencer

Erik Hubert Dolly Goossens

Saurabh Singh

Ruofei Du

Vladimir Tankovich

Graham Fyffe

Steven L Goldberg

Steven Benjamin Goldberg

Yun-Ta Tsai

Matthew Alun Brown

Atulit Kumar

Julien Pascal Christophe Valentin

Nidhi Rathi

Daniel William Erickson

Jessica Lynn Busch

Navin Sarma

Xueming Yu

Sergio Orts Escolano

Jason Angelo Dourgarian

Eric Aboussouan

Deqing Sun

Moustafa Meshry

Matthew Whalen

Geoffrey Douglas Harvey

Peter Christopher Lincoln

Hossam Isack

Ivana Tosic Rodgers

Grant Yoshida

Qinge Wu

Abhimitra Meka

Feitong Tan

Peter Barnum

Abhi Meka

Phillip Andrew Chou

Catherine Boulanger

Peter Joseph Denny

Kaiwen Guo

Michael Milne

Ryan Overbeck

Thabo Beeler

Christian Häne

Sarah Amsellem

Matthew Brown

Example embodiments relate to techniques for volumetric performance capture with neural rendering. A technique may involve initially obtaining images that depict a subject from multiple viewpoints and under various lighting conditions using a light stage and depth data corresponding to the subject using infrared cameras. A neural network may extract features of the subject from the images based on the depth data and map the features into a texture space (e.g., the UV texture space). A neural renderer can be used to generate an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view. The neural render may resample the features of the subject from the texture space to an image space to generate the output image.

Volumetric performance capture with neural rendering

Systems and methods are directed to encoding and/or decoding of the textures/geometry of a three-dimensional volumetric representation. An encoding computing system can obtain voxel blocks from a three-dimensional volumetric representation of an object. The encoding computing system can encode voxel blocks with a machine-learned voxel encoding model to obtain encoded voxel blocks. The encoding computing system can decode the encoded voxel blocks with a machine-learned voxel decoding model to obtain reconstructed voxel blocks. The encoding computing system can generate a reconstructed mesh representation of the object based at least in part on the one or more reconstructed voxel blocks. The encoding computing system can encode textures associated with the voxel blocks according to an encoding scheme and based at least in part on the reconstructed mesh representation of the object to obtain encoded textures.

Systems and methods for compression of three-dimensional volumetric representations

Apparatus and methods related to applying lighting models to images of objects are provided. An example method includes applying a geometry model to an input image to determine a surface orientation map indicative of a distribution of lighting on an object based on a surface geometry. The method further includes applying an environmental light estimation model to the input image to determine a direction of synthetic lighting to be applied to the input image. The method also includes applying, based on the surface orientation map and the direction of synthetic lighting, a light energy model to determine a quotient image indicative of an amount of light energy to be applied to each pixel of the input image. The method additionally includes enhancing, based on the quotient image, a portion of the input image. One or more neural networks can be trained to perform one or more of the aforementioned aspects.

Enhanced photo relighting based on machine learning models

Techniques of identifying gestures include detecting and classifying inner-wrist muscle motions at a user's wrist using micron-resolution radar sensors. For example, a user of an AR system may wear a band around their wrist. When the user makes a gesture to manipulate a virtual object in the AR system as seen in a head-mounted display (HMD), muscles and ligaments in the user's wrist make small movements on the order of 1-3 mm. The band contains a small radar device that has a transmitter and a number of receivers (e.g., three) of electromagnetic (EM) radiation on a chip (e.g., a Soli chip. This radiation reflects off the wrist muscles and ligaments and is received by the receivers on the chip in the band. The received reflected signal, or signal samples, are then sent to processing circuitry for classification to identify the wrist movement as a gesture.

Hand gesture recognition based on detected wrist muscular movements

Systems and methods are described for utilizing an image processing system with at least one processing device to perform operations including receiving a plurality of input images of a user, generating a three-dimensional mesh proxy based on a first set of features extracted from the plurality of input images and a second set of features extracted from the plurality of input images. The method may further include generating a neural texture based on a three-dimensional mesh proxy and the plurality of input images, generating a representation of the user including at least a neural texture, and sampling at least one portion of the neural texture from the three-dimensional mesh proxy. In response to providing the at least one sampled portion to a neural renderer, the method may include receiving, from the neural renderer, a synthesized image of the user that is previously not captured by the image processing system.

Few-shot synthesis of talking heads

A head-mounted device (HMD) can be configured to determine a request for recognizing at least one content item included within content framed within a display of the HMD. The HMD can be configured to initiate a head-tracking process that maintains a coordinate system with respect to the content, and a pointer-tracking process that tracks a pointer that is visible together with the content within the display. The HMD can be configured to capture a first image of the content and a second image of the content, the second image including the pointer. The HMD can be configured to map a location of the pointer within the second image to a corresponding image location within the first image, using the coordinate system, and provide the at least one content item from the corresponding image location.

Pointer-based content recognition using a head-mounted device

Methods, systems, and apparatus including computer programs encoded on a computer storage medium, for generating convex decomposition of objects using neural network models. One of the methods includes receiving an input that depicts an object. The input is processed using a neural network to generate an output that defines a convex representation of the object. The output includes, for each of a plurality of convex elements, respective parameters that define a position of the convex element in the convex representation of the object.

Convex representation of objects using neural network

Systems and methods for training models to predict dense correspondences across images such as human images. A model may be trained using synthetic training data created from one or more 3D computer models of a subject. In addition, one or more geodesic distances derived from the surfaces of one or more of the 3D models may be used to generate one or more loss values, which may in turn be used in modifying the model's parameters during training.

Systems and methods for training models to predict dense correspondences in images using geodesic distances

Techniques of tracking a user's gaze includes identifying a region of a display at which a gaze of a user is directed, the region including a plurality of pixels. By determining a region rather than a point, when the regions correspond to elements of a user interface, the improved technique enables a system to activate the element to which a determined region is selected. In some implementations, the system makes the determination using a classification engine including a convolutional neural network; such an engine takes as input images of the user's eye and outputs a list of probabilities that the gaze is directed to each of the regions.

Eye gaze classification

A method of animating a digital character according to facial expressions of a user, comprising the steps of, (a) obtaining a 2D image and 3D depth map of the face of the user, (b) determining expression parameters for a user expression model so that a facial expression of the user-specific expression model represents the face of the user shown in the 2D image and 3D depth map (c) using the expression parameters and an animation prior to determine animation parameters usable to animate a digital character, wherein the animation prior is a sequence of animation parameters which represent predefined animations of a digital character (d) using the animation parameters to animate a digital character so that the digital character mimics the face of the user.

Method for facial animation

According to an aspect, a real-time active stereo system includes a capture system configured to capture stereo data, where the stereo data includes a first input image and a second input image, and a depth sensing computing system configured to predict a depth map. The depth sensing computing system includes a feature extractor configured to extract features from the first and second images at a plurality of resolutions, an initialization engine configured to generate a plurality of depth estimations, where each of the plurality of depth estimations corresponds to a different resolution, and a propagation engine configured to iteratively refine the plurality of depth estimations based on image warping and spatial propagation.

Real-time stereo matching using a hierarchical iterative refinement network

Systems and methods are described for generating a plurality of three-dimensional (3D) proxy geometries of an object, generating, based on the plurality of 3D proxy geometries, a plurality of neural textures of the object, the neural textures defining a plurality of different shapes and appearances representing the object, providing the plurality of neural textures to a neural renderer, receiving, from the neural renderer and based on the plurality of neural textures, a color image and an alpha mask representing an opacity of at least a portion of the object, and generating a composite image based on the pose, the color image, and the alpha mask.

Generative latent textured proxies for object category modeling

A wearable computing device includes a frame, a camera mounted on the frame so as to capture images of an environment outside of the wearable computing device, a display device mounted on the frame so as to display the images captured by the camera, and at least one eye gaze tracking device mounted on the frame so as to track a gaze directed at the images displayed by the display device. In response to the detection of a fixation of the gaze on the display of images, the system may identify a pixel area corresponding to a fixation point of the fixation gaze on the display of images. The system may identify an object in the ambient environment corresponding to the identified pixel area, and set the identified object as a selected object for user interaction.

Object selection based on eye tracking in wearable device

Growing community of inventors

Los Gatos, CA, United States of America

Sofien Bouaziz