Microsoft Technology Licensing, LLC

Jonathan C Hanzelka

Pedro Urbina Escos

Ishani Chakraborty

Thomas M Soemo

Michael Ebstyne

Emanuel Shalev

Lu Yuan

Trebor Lee Connell

Gavin Jancke

Muhammad Zeeshan Zia

Harpreet Singh Sawhney

Vibhav Vineet

Vicente A Rivera

A method for virtual object pose rendering includes receiving a digital image of a real-world object in a real-world environment. An imaged pose of the real-world object is estimated based at least in part on a neural radiance model encoding 3D (three-dimensional) spatial data for the real-world object, the neural radiance model derived from a plurality of previously captured digital images of the real-world object. A view of a virtual object corresponding to the real-world object is rendered based at least in part on the neural radiance model, such that the virtual object has a virtual pose consistent with the imaged pose of the real-world object in the digital image.

Virtual pose rendering using neural radiance model

Training a multi-object tracking model includes: generating a plurality of training images based at least on scene generation information, each training image comprising a plurality of objects to be tracked; generating, for each training image, original simulated data based at least on the scene generation information, the original simulated data comprising tag data for a first object; locating, within the original simulated data, tag data for the first object, based on at least an anomaly alert (e.g., occlusion alert, proximity alert, motion alert) associated with the first object in the first training image; based at least on locating the tag data for the first object, modifying at least a portion of the tag data for the first object from the original simulated data, thereby generating preprocessed training data from the original simulated data; and training a multi-object tracking model with the preprocessed training data to produce a trained multi-object tracker.

Training multi-object tracking models using simulation

Systems and methods are disclosed for leveraging rendering engines to perform multi-spectral rendering by reusing the color channels for additional spectral bands. A digital asset represented by a three dimensional (3D) mesh and a material reference pointer may be rendered using a first material spectral band data set and additionally rendered using a second material spectral band data set, and the results combined to create a multi-spectral rendering. The multi-spectral rendering may then be used as part of a synthetics service or operation. By abstracting the material properties, a material translator is able to return a banded material data set from among a plurality of spectral band sets, and asset material information may advantageously be managed apart from managing each asset individually.

Multi-spectral rendering for synthetics

The disclosure herein describes training a machine learning model to recognize a real-world object based on generated virtual scene variations associated with a model of the real-world object. A digitized three-dimensional (3D) model representing the real-world object is obtained and a virtual scene is built around the 3D model. A plurality of virtual scene variations is generated by varying one or more characteristics. Each virtual scene variation is generated to include a label identifying the 3D model in the virtual scene variation. A machine learning model may be trained based on the plurality of virtual scene variations. The use of generated digital assets to train the machine learning model greatly decreases the time and cost requirements of creating training assets and provides training quality benefits based on the quantity and quality of variations that may be generated, as well as the completeness of information included in each generated digital asset.

Generating synthetic digital assets for a virtual scene including a model of a real-world object

Growing community of inventors

Kenmore, WA, United States of America

Jonathan C Hanzelka