Microsoft Technology Licensing, LLC

Stephan Joachim Garbin

Matthew Alastair Johnson

Tadas Baltrusaitis

Marek Adam Kowalski

Virginia Estellers Casas

Martin De La Gorce

Sebastian Karol Dziadzio

Jamie Daniel Joseph Shotton

Julien Pascal Christophe Valentin

Jingjing Shen

Daniel Stephen Wilde

Toby Leonard Sharp

Thomas Joseph Cashman

Erroll William Wood

Christopher Maurice Mei

Charles Thomas Hewitt

Marta Malgorzata Wilczkowiak

Paul Malcolm McIlroy

Nikola Milosavljevic

Ivan Stojiljkovic

Shideh Rezaeifar

Kacper Kania

Stanislaw Kacper Szymanowicz

There is a region of interest of a synthetic image depicting an object from a class of objects. A trained neural image generator, having been trained to map embeddings from a latent space to photorealistic images of objects in the class, is accessed. A first embedding is computed from the latent space, the first embedding corresponding to an image which is similar to the region of interest while maintaining photorealistic appearance. A second embedding is computed from the latent space, the second embedding corresponding to an image which matches the synthetic image. Blending of the first embedding and the second embedding is done to form a blended embedding. At least one output image is generated from the blended embedding, the output image being more photorealistic than the synthetic image.

Computing photorealistic versions of synthetic images

Keypoints are predicted in an image. Predictions are generated for each of the keypoints of an image as a 2D random variable, normally distributed with location (x, y) and standard deviation sigma. A neural network is trained to maximize a log-likelihood that samples from each of the predicted keypoints equal a ground truth. The trained neural network is used to predict keypoints of an image without generating a heatmap.

Model fitting using keypoint regression

A cage of primitive 3D elements and associated animation data is received. Compute a ray from a virtual camera through a pixel into the cage animated according to the animation data and compute a plurality of samples on the ray. Compute a transformation of the samples into a canonical cage. For each transformed sample, query a plurality of learnt radiance field parameterizations, each learnt on a different deformed state of the 3D scene to obtain color values for each learnt radiance field. For each transformed sample, query a learnt radiance field parameterization of the 3D scene to obtain an opacity value. Compute, for each transformed sample, a weighted combination of the color values, wherein the weights are related to the local features. A volume rendering method is applied to the weighted combinations of the color and the opacity values producing a pixel value.

Dynamic 3D scene generation

To compute an image of a dynamic 3D scene comprising a 3D object, a description of a deformation of the 3D object is received, the description comprising a cage of primitive 3D elements and associated animation data from a physics engine or an articulated object model. For a pixel of the image the method computes a ray from a virtual camera through the pixel into the cage animated according to the animation data and computes a plurality of samples on the ray. Each sample is a 3D position and view direction in one of the 3D elements. The method computes a transformation of the samples into a canonical cage. For each transformed sample, the method queries a learnt radiance field parameterization of the 3D scene to obtain a color value and an opacity value. A volume rendering method is applied to the color and opacity values producing a pixel value of the image.

Computing images of controllable dynamic scenes

In various examples there is an apparatus for computing an image depicting a face of a wearer of a head mounted display (HMD), as if the wearer was not wearing the HMD. An input image depicts a partial view of the wearer's face captured from at least one face facing capture device in the HMD. A machine learning apparatus is available which has been trained to compute expression parameters from the input image. A 3D face model that has expressions parameters is accessible as well as a photorealiser being a machine learning model trained to map images rendered from the 3D face model to photorealistic images. The apparatus computes expression parameter values from the image using the machine learning apparatus. The apparatus drives the 3D face model with the expression parameter values to produce a 3D model of the face of the wearer and then renders the 3D model from a specified viewpoint to compute a rendered image. The rendered image is upgraded to a photorealistic image using the photorealiser.

Computing images of head mounted display wearer

In various examples there is a method of image processing comprising: storing a real image of an object in memory, the object being a specified type of object. The method involves computing, using a first encoder, a factorized embedding of the real image. The method receives a value of at least one parameter of a synthetic image rendering apparatus for rendering synthetic images of objects of the specified type. The parameter controls an attribute of synthetic images of objects rendered by the rendering apparatus. The method computes an embedding factor of the received value using a second encoder. The factorized embedding is modified with the computed embedding factor. The method computes, using a decoder with the modified embedding as input, an output image of an object which is substantially the same as the real image except for the attribute controlled by the parameter.

Controllable image generation

Methods and systems are provided for training a machine learning model to generate density values and radiance components based on positional data, along with a weighting scheme associated with a particular view direction based on directional data to compute a final RGB value for each point along a plurality of camera rays. The positional data and directional data are extracted from set of training images of a particular static scene. The radiance components, density values, and weighting schemes are cached for efficient image data processing to perform volume rendering for each point sampled. A novel viewpoint of a static scene is generated based on the volume rendering for each point sampled.

High resolution neural rendering

Growing community of inventors

London, United Kingdom

Stephan Joachim Garbin