Microsoft Technology Licensing, LLC

Jingjing Shen

Thomas Joseph Cashman

Erroll William Wood

Toby Leonard Sharp

Julien Pascal Christophe Valentin

Matthew Alastair Johnson

Tadas Baltrusaitis

Stephan Joachim Garbin

Timothy James Hutton

Jamie Daniel Joseph Shotton

Andrew William Fitzgibbon

Jamie Shotton

Charles Thomas Hewitt

Ivan Razumenic

Virginia Estellers Casas

Kenneth Mitchell Jakubzak

Brent Michael Wilson

Martin De La Gorce

Daniel Stephen Wilde

Predrag Rodoljub Jovanovic

Marek Adam Kowalski

Matthew Julian Lamb

Nikola Milosavljevic

Ivan Stojiljkovic

Shideh Rezaeifar

Stanislaw Kacper Szymanowicz

Keypoints are predicted in an image. Predictions are generated for each of the keypoints of an image as a 2D random variable, normally distributed with location (x, y) and standard deviation sigma. A neural network is trained to maximize a log-likelihood that samples from each of the predicted keypoints equal a ground truth. The trained neural network is used to predict keypoints of an image without generating a heatmap.

Model fitting using keypoint regression

The techniques described herein disclose a system that is configured to detect and track the three-dimensional pose of an object (e.g., a head-mounted display device) in a color image using an accessible three-dimensional model of the object. The system uses the three-dimensional pose of the object to repair pixel depth values associated with a region (e.g., a surface) of the object that is composed of material that absorbs light emitted by a time-of-flight depth sensor to determine depth. Consequently, a color-depth image (e.g., a Red-Green-Blue-Depth image or RGB-D image) can be produced that does not include dark holes on and around the region of the object that is composed of material that absorbs light emitted by the time-of-flight depth sensor.

Repairing image depth values for an object with a light absorbing surface

To compute an image of a dynamic 3D scene comprising a 3D object, a description of a deformation of the 3D object is received, the description comprising a cage of primitive 3D elements and associated animation data from a physics engine or an articulated object model. For a pixel of the image the method computes a ray from a virtual camera through the pixel into the cage animated according to the animation data and computes a plurality of samples on the ray. Each sample is a 3D position and view direction in one of the 3D elements. The method computes a transformation of the samples into a canonical cage. For each transformed sample, the method queries a learnt radiance field parameterization of the 3D scene to obtain a color value and an opacity value. A volume rendering method is applied to the color and opacity values producing a pixel value of the image.

Computing images of controllable dynamic scenes

In various examples there is an apparatus for aligning three-dimensional, 3D, representations of people. The apparatus comprises at least one processor and a memory storing instructions that, when executed by the at least one processor, perform a method comprising accessing a first 3D representation which is an instance of a parametric model of a person; accessing a second 3D representation which is a photoreal representation of the person; computing an alignment of the first and second 3D representations; and computing and storing a hologram from the aligned first and second 3D representations such that the hologram depicts parts of the person which are observed in only one of the first and second 3D representations; or controlling an avatar representing the person where the avatar depicts parts of the person which are observed in only one of the first and second 3D representations.

Alignment of 3D representations for hologram/avatar control

A tracker is described which comprises a processor configured to receive captured sensor data depicting an object. The processor is configured to access a rigged polygon mesh model of the object and to compute a plurality of approximate surface normals of a limit surface of the rigged polygon mesh. The processor is configured to compute values of pose parameters of the model by calculating an optimization to fit the model to the captured sensor data where the optimization uses an evaluation function based on the plurality of approximate surface normals.

Tracking rigged polygon-mesh models of articulated objects

In various examples there is an apparatus for detecting position and orientation of an object. The apparatus comprises a memory storing at least one frame of captured sensor data depicting the object. The apparatus also comprises a trained machine learning system configured to receive the frame of the sensor data and to compute a plurality of two dimensional positions in the frame. Each predicted two dimensional position is a position of sensor data in the frame depicting a keypoint, where a keypoint is a pre-specified 3D position relative to the object. At least one of the keypoints is a floating keypoint depicting a pre-specified position relative to the object, lying inside or outside the object's surface. The apparatus comprises a pose detector which computes the three dimensional position and orientation of the object using the predicted two dimensional positions and outputs the computed three dimensional position and orientation.

Detecting pose using floating keypoint(s)

Growing community of inventors

Cambridge, United Kingdom

Jingjing Shen