Microsoft Technology Licensing, LLC

Baoyuan Wang

Sing Bing Kang

Joshua Bryan Weisberg

Noranart Vesdapunt

Baining Guo

Hugues Herve Hoppe

Neel Suresh Joshi

Stephen S Lin

Szymon Piotr Stachniak

Zhuowen Tu

Ke Deng

Gaurav Mittal

Wei-Chih Chen

Ying Jin

Gil Moshe Nahmias

Hsiang-Tao Wu

Tae-Hyun Oh

Wenbin Zhu

Pierrick Arsenault

Zeyu Chen

Utkarsh Sinha

Kandarpkumar J Makwana

Yuanming Hu

Melissa Regalia

Gil M Nahmias

Techniques performed by a data processing system for reconstructing a three-dimensional (3D) model of the face of a human subject herein include obtaining source data comprising a two-dimensional (2D) image, three-dimensional (3D) image, or depth information representing a face of a human subject. Reconstructing the 3D model of the face also includes generating a 3D model of the face of the human subject based on the source data by analyzing the source data to produce a coarse 3D model of the face of the human subject, and refining the coarse 3D model through free form deformation to produce a fitted 3D model. The coarse 3D model may be a 3D Morphable Model (3DMM), and the coarse 3D model may be refined through free-form deformation in which the deformation of the mesh is limited by applying an as-rigid-as-possible (ARAP) deformation constraint.

Reinforced differentiable attribute for 3D face reconstruction

A computer-implemented technique for animating a visual representation of a face based on spoken words of a speaker is described herein. A computing device receives an audio sequence comprising content features reflective of spoken words uttered by a speaker. The computing device generates latent content variables and latent style variables based upon the audio sequence. The latent content variables are used to synchronized movement of lips on the visual representation to the spoken words uttered by the speaker. The latent style variables are derived from an expected appearance of facial features of the speaker as the speaker utters the spoken words and are used to synchronize movement of full facial features of the visual representation to the spoken words uttered by the speaker. The computing device causes the visual representation of the face to be animated on a display based upon the latent content variables and the latent style variables.

Computing system for expressive three-dimensional facial animation

Described herein is a human pose prediction system and method. An image comprising at least a portion of a human body is received. A trained neural network is used to predict one or more human features (e.g., joints/aspects of a human body) within the received image, and, to predict one or more human poses in accordance with the predicted one or more human features. The trained neural network can be an end-to-end trained, single stage deep neural network. An action is performed based on the predicted one or more human poses. For example, the human pose(s) can be displayed as an overlay with received image.

Human pose estimation

Disclosed in some examples, are methods, systems, and machine readable mediums that correct image color casts by utilizing a fully convolutional network (FCN), where the patches in an input image may differ in influence over the color constancy estimation. This influence is formulated as a confidence weight that reflects the value of a patch for inferring the illumination color. The confidence weights are integrated into a novel pooling layer where they are applied to local patch estimates in determining a global color constancy result.

Fully convolutional color constancy with confidence weighted pooling

A 'Best of Burst Selector,' or 'BoB Selector,' automatically selects a subjectively best image from a single set of images of a scene captured in a burst or continuous capture mode, captured as a video sequence, or captured as multiple images of the scene over any arbitrary period of time and any arbitrary timing between images. This set of images is referred to as a burst set. Selection of the subjectively best image is achieved in real-time by applying a machine-learned model to the burst set. The machine-learned model of the BoB Selector is trained to select one or more subjectively best images from the burst set in a way that closely emulates human selection based on subjective subtleties of human preferences. Images automatically selected by the BoB Selector are presented to a user or saved for further processing.

Automated selection of subjectively best image frames from burst captured image sequences

An 'Exposure Controller' provides various techniques for training and applying a deep convolution network to provide real-time automated camera exposure control, as a real-time function of scene semantic context, in a way that improves image quality for a wide range of image subject types in a wide range of real-world lighting conditions. The deep learning approach applied by the Exposure Controller to implement this functionality first uses supervised learning to achieve a good anchor point that mimics integral exposure control for a particular camera model or type, followed by refinement through reinforcement learning. The end-to-end system (e.g., exposure control and image capture) provided by the Exposure Controller provides real-time performance for predicting and setting camera exposure values to improve overall visual quality of the resulting image over a wide range of image capture scenarios (e.g., back-lit scenes, front lighting, rapid changes to lighting conditions, etc.).

Real-time semantic-aware camera exposure control

Techniques for automatically selecting image frames from a video and providing the selected image frames to a device for display are disclosed.

Deep learning on image frames to generate a summary

Low-latency gesture detection is described, for example, to compute a gesture class from a live stream of image frames of a user making a gesture, for example, as part of a natural user interface controlling a game system or other system. In examples, machine learning components are trained to learn gesture primitives and at test time, are able to detect gestures using the learned primitives, in a fast, accurate manner. For example, a gesture primitive is a latent (unobserved) variable features of a subset of frames from a sequence of frames depicting a gesture. For example, the subset of frames has many fewer frames than a sequence of frames depicting a complete gesture. In various examples gesture primitives are learnt from instance level features computed by aggregating frame level features to capture temporal structure. In examples frame level features comprise body position and body part articulation state features.

Low-latency gesture detection

A 'Moment Capture System' automatically captures and buffers multiple image frames, both prior to and following digital camera shutter activation. This capture and buffering is performed in combination with real-time automatic control of camera settings (e.g., exposure time, capture rate, ISO, white balance, aperture, etc.) based on an ongoing real-time evaluation of contents and characteristics of most recent previously buffered frames. Whenever a shutter activation (e.g., a 'tap') is detected, the Moment Capture System pulls some number of pre-tap frames from the buffer and adds some number of post-tap frames to create an &#x201c;image moment.&#x201d; Image moments are defined as sets of sequential frames spanning a time period before and after the detected tap. In various implementations, the Moment Capture System performs automated selection of perceptually best images from the buffered frames associated with each tap. This automated selection closely emulates human selection based on subjective subtleties of human preferences.

Variable pre- and post-shot continuous frame buffering with automated image selection and enhancement

A cinemagraph is generated that includes one or more video loops. A cinemagraph generator receives an input video, and semantically segments the frames to identify regions that correspond to semantic objects and the semantic object depicted in each identified region. Input time intervals are then computed for the pixels of the frames of the input video. An input time interval for a particular pixel includes a per-pixel loop period and a per-pixel start time of a loop at the particular pixel. In addition, the input time interval of a pixel is based, in part, on one or more semantic terms which keep pixels associated with the same semantic object in the same video loop. A cinemagraph is then created using the input time intervals computed for the pixels of the frames of the input video.

Growing community of inventors

Sammamish, WA, United States of America

Baoyuan Wang