Microsoft Technology Licensing, LLC

Toby Leonard Sharp

Jamie Daniel Joseph Shotton

Andrew William Fitzgibbon

Pushmeet Kohli

Antonio Criminisi

Shahram Izadi

Carsten Curt Eckard Rother

Reinhard Sebastian Bernhard Nowozin

Cem Keskin

Thomas Joseph Cashman

Duncan Paul Robertson

Andrew Blake

Erroll William Wood

Vladimir Kolmogorov

Jingjing Shen

Jonathan James Taylor

John Michael Winn

Jonathan Taylor

Christoph Rhemann

Tadas Baltrusaitis

Julien Pascal Christophe Valentin

Matthew Darius Cook

Matthew Alastair Johnson

Jamie Shotton

Stephan Joachim Garbin

Daniel Stefan Tarlow

Ivan Razumenic

Kenneth Mitchell Jakubzak

Charles Thomas Hewitt

Khan Mohammad Siddiqui

Brent Michael Wilson

Martin De La Gorce

Daniel Stephen Wilde

Benjamin Michael Glocker

Predrag Rodoljub Jovanovic

Bangpeng Yao

Evgeny Salnikov

Matthew Julian Lamb

Nikola Milosavljevic

Ivan Stojiljkovic

David Joseph New Tan

Sameh Khamis

Abner Guzmán-Rivera

Keypoints are predicted in an image. Predictions are generated for each of the keypoints of an image as a 2D random variable, normally distributed with location (x, y) and standard deviation sigma. A neural network is trained to maximize a log-likelihood that samples from each of the predicted keypoints equal a ground truth. The trained neural network is used to predict keypoints of an image without generating a heatmap.

Model fitting using keypoint regression

The techniques described herein disclose a system that is configured to detect and track the three-dimensional pose of an object (e.g., a head-mounted display device) in a color image using an accessible three-dimensional model of the object. The system uses the three-dimensional pose of the object to repair pixel depth values associated with a region (e.g., a surface) of the object that is composed of material that absorbs light emitted by a time-of-flight depth sensor to determine depth. Consequently, a color-depth image (e.g., a Red-Green-Blue-Depth image or RGB-D image) can be produced that does not include dark holes on and around the region of the object that is composed of material that absorbs light emitted by the time-of-flight depth sensor.

Repairing image depth values for an object with a light absorbing surface

In various examples there is an apparatus for aligning three-dimensional, 3D, representations of people. The apparatus comprises at least one processor and a memory storing instructions that, when executed by the at least one processor, perform a method comprising accessing a first 3D representation which is an instance of a parametric model of a person; accessing a second 3D representation which is a photoreal representation of the person; computing an alignment of the first and second 3D representations; and computing and storing a hologram from the aligned first and second 3D representations such that the hologram depicts parts of the person which are observed in only one of the first and second 3D representations; or controlling an avatar representing the person where the avatar depicts parts of the person which are observed in only one of the first and second 3D representations.

Alignment of 3D representations for hologram/avatar control

Memory facilitation using directed acyclic graphs is described, for example, where a plurality of directed acyclic graphs are trained for gesture recognition from human skeletal data, or to estimate human body joint positions from depth images for gesture detection. In various examples directed acyclic graphs are grown during training using a training objective which takes into account both connection patterns between nodes and split function parameter values. For example, a layer of child nodes is grown and connected to a parent layer of nodes using an initialization strategy. In examples, various local search processes are used to find good combinations of connection patterns and split function parameters.

Memory facilitation using directed acyclic graphs

Region of interest detection in raw time of flight images is described. For example, a computing device receives at least one raw image captured for a single frame by a time of flight camera. The raw image depicts one or more objects in an environment of the time of flight camera (such as human hands, bodies or any other objects). The raw image is input to a trained region detector and in response one or more regions of interest in the raw image are received. A received region of interest comprises image elements of the raw image which are predicted to depict at least part of one of the objects. A depth computation logic computes depth from the one or more regions of interest of the raw image.

Depth from time of flight camera

A tracker is described which comprises an input configured to receive captured sensor data depicting an object. The tracker has a processor configured to access a rigged, smooth-surface model of the object and to compute values of pose parameters of the model by calculating an optimization to fit the model to data related to the captured sensor data. Variables representing correspondences between the data and the model are included in the optimization jointly with the pose parameters.

Tracking rigged smooth-surface models of articulated objects

Model fitting from raw time of flight image data is described, for example, to track position and orientation of a human hand or other entity. In various examples, raw image data depicting the entity is received from a time of flight camera. A 3D model of the entity is accessed and used to render, from the 3D model, simulations of raw time of flight image data depicting the entity in a specified pose/shape. The simulated raw image data and at least part of the received raw image data are compared and on the basis of the comparison, parameters of the entity are computed.

Model fitting from raw time-of-flight images

Examples describe an apparatus for calibrating a three dimensional (3D) mesh model of an articulated object. The articulated object is an instance of a specified object class. The apparatus comprises an input configured to receive captured sensor data depicting the object. The apparatus has a calibration engine configured to compute values of shape parameters of the 3D mesh model which indicate which member of the object class is depicted in the captured sensor data, in order to calibrate the 3D mesh model. The calibration engine is configured to compute the values of the shape parameters with an optimization process to find at least one potential local or global minimum of an energy function, the energy function expressing a degree of similarity between data rendered from the model and the received sensor data.

Calibrating object shape

Tracking hand or body pose from image data is described, for example, to control a game system, natural user interface or for augmented reality. In various examples a prediction engine takes a single frame of image data and predicts a distribution over a pose of a hand or body depicted in the image data. In examples, a stochastic optimizer has a pool of candidate poses of the hand or body which it iteratively refines, and samples from the predicted distribution are used to replace some candidate poses in the pool. In some examples a best candidate pose from the pool is selected as the current tracked pose and the selection processes uses a 3D model of the hand or body.

Tracking hand/body pose

Tracking using sensor data is described, for example, where a plurality of machine learning predictors are used to predict a plurality of complementary, or diverse, parameter values of a process describing how the sensor data arises. In various examples a selector selects which of the predicted values are to be used, for example, to control a computing device. In some examples the tracked parameter values are pose of a moving camera or pose of an object moving in the field of view of a static camera; in some examples the tracked parameter values are of a 3D model of a hand or other articulated or deformable entity. The machine learning predictors have been trained in series, with training examples being reweighted after training an individual predictor, to favor training examples on which the set of predictors already trained performs poorly.

Tracking using sensor data

Depth image compression is described for example, to enable body-part centers of players of a game to be detected in real time from depth images or for other applications such as augmented reality, and human-computer interaction. In an embodiment, depth images which have associated body-part probabilities, are compressed using probability mass which is related to the depth of an image element and a probability of a body part for the image element. In various examples, compression of the depth images using probability mass enables body part center detection, by clustering output elements, to be speeded up. In some examples, the scale of the compression is selected according to a depth of a foreground region and in some cases different scales are used for different image regions. In some examples, certainties of the body-part centers are calculated using probability masses of clustered image elements.

Depth image compression

Foreground and background image segmentation is described. In an example, a seed region is selected in a foreground portion of an image, and a geodesic distance is calculated from each image element to the seed region. A subset of the image elements having a geodesic distance less than a threshold is determined, and this subset of image elements are labeled as foreground. In another example, an image element from an image showing at least a user, a foreground object in proximity to the user, and a background is applied to trained decision trees to obtain probabilities of the image element representing one of these items, and a corresponding classification assigned to the image element. This is repeated for each image element. Image elements classified as belonging to the user are labeled as foreground, and image elements classified as foreground objects or background are labeled as background.

Foreground and background image segmentation

Growing community of inventors

Cambridge, United Kingdom

Toby Leonard Sharp