Microsoft Technology Licensing, LLC

Michael Ebstyne

Pedro Urbina Escos

Emanuel Shalev

Alex A Kipman

Yuri Pekelny

Ethan Eade

Drew Edward Steedly

Kamran Rajabi Zargahi

Stephen Michelotti

Jonathan Chi Hang Chan

Frederik Schaffalitzky

Adam Smolinski

Trebor Lee Connell

Jonathan C Hanzelka

Michael Grabner

Georg Klein

Mark Flick

Stephen G Latta

Larry Marvin Wall

Martin Thomas Shetter

Muhammad Zeeshan Zia

Paul Albert Lalonde

Harpreet Singh Sawhney

Calvin Chan

Don Dongwoo Kim

Alvin Wen Jie Chia

Frederick Schaffalitzky

A synthetic world interface may be used to model digital environments, sensors, and motions for the evaluation, development, and improvement of localization algorithms. A synthetic data cloud service with a library of sensor primitives, motion generators, and environments with procedural and game-like capabilities, facilitates engineering design for a manufactural solution that has localization capabilities. In some embodiments, a sensor platform simulator operates with a motion orchestrator, an environment orchestrator, an experiment generator, and an experiment runner to test various candidate hardware configurations and localization algorithms in a virtual environment, advantageously speeding development and reducing cost. Thus, examples disclosed herein may relate to virtual reality (VR) or mixed reality (MR) implementations.

Spatial localization design service

A synthetic world interface may be used to model digital environments, sensors, and motions for the evaluation, development, and improvement of computer vision and speech algorithms. A synthetic data cloud service with a library of sensor primitives, motion generators, and environments with procedural and game-like capabilities, facilitates engineering design for a manufactural solution that has computer vision and speech capabilities. In some embodiments, a sensor platform simulator operates with a motion orchestrator, an environment orchestrator, an experiment generator, and an experiment runner to test various candidate hardware configurations and computer vision and speech algorithms in a virtual environment, advantageously speeding development and reducing cost. Thus, examples disclosed herein may relate to virtual reality (VR) or mixed reality (MR) implementations.

Computer vision and speech algorithm design service

Various embodiments, methods and systems for implementing a distributed computing system crowdsourcing engine are provided. Initially, a source asset is received from a distributed synthetic data as a service (SDaaS) crowdsource interface. A crowdsource tag is received for the source asset via the distributed SDaaS crowdsource interface. Based in part on the crowdsource tag, the source asset is ingested. Ingesting the source asset comprises automatically computing values for asset-variation parameters of the source asset. The asset-variation parameters are programmable for machine-learning. A crowdsourced synthetic data asset comprising the values for asset-variation parameters is generated.

Distributed computing system with a crowdsourcing engine

Various embodiments, methods and systems for implementing a distributed computing system scene assembly engine are provided. Initially, a selection of a first synthetic data asset and a selection of a second synthetic data asset are received from a distributed synthetic data as a service (SDaaS) integrated development environment (IDE). A synthetic data asset is associated with asset-variation parameters and scene-variation parameters, the asset-variation parameters and scene-variation parameters are programmable for machine-learning. Values for generating a synthetic data scene are received. The values correspond to asset-variation parameters or scene-variation parameters. Based on the values, the synthetic data scene is generated using the first synthetic data asset and the second synthetic data asset.

Distributed computing system with a synthetic data as a service scene assembly engine

Systems and methods are disclosed for leveraging rendering engines to perform multi-spectral rendering by reusing the color channels for additional spectral bands. A digital asset represented by a three dimensional (3D) mesh and a material reference pointer may be rendered using a first material spectral band data set and additionally rendered using a second material spectral band data set, and the results combined to create a multi-spectral rendering. The multi-spectral rendering may then be used as part of a synthetics service or operation. By abstracting the material properties, a material translator is able to return a banded material data set from among a plurality of spectral band sets, and asset material information may advantageously be managed apart from managing each asset individually.

Multi-spectral rendering for synthetics

Various embodiments, methods and systems for implementing a distributed computing system feedback loop engine are provided. Initially, a training dataset report is accessed. The training dataset report identifies a synthetic data asset having values for asset-variation parameters. The synthetic data asset is associated with a frameset. Based on the training dataset report, the synthetic data asset with a synthetic data asset variation is updated. The frameset is updated using the updated synthetic data asset.

Distributed computing system with a synthetic data as a service feedback loop engine

An immersive feedback loop is disclosed for improving artificial intelligence (AI) applications used for virtual reality (VR) environments. Users may iteratively generate synthetic scene training data, train a neural network on the synthetic scene training data, generate synthetic scene evaluation data for an immersive VR experience, indicate additional training data needed to correct neural network errors indicated in the VR experience, and then generate and retrain on the additional training data, until the neural network reaches an acceptable performance level.

Immersive feedback loop for improving AI

The disclosure herein describes training a machine learning model to recognize a real-world object based on generated virtual scene variations associated with a model of the real-world object. A digitized three-dimensional (3D) model representing the real-world object is obtained and a virtual scene is built around the 3D model. A plurality of virtual scene variations is generated by varying one or more characteristics. Each virtual scene variation is generated to include a label identifying the 3D model in the virtual scene variation. A machine learning model may be trained based on the plurality of virtual scene variations. The use of generated digital assets to train the machine learning model greatly decreases the time and cost requirements of creating training assets and provides training quality benefits based on the quantity and quality of variations that may be generated, as well as the completeness of information included in each generated digital asset.

Generating synthetic digital assets for a virtual scene including a model of a real-world object

Various embodiments, methods and systems for implementing a distributed computing frameset assembly engine are provided. Initially, a synthetic data scene is accessed. A first set of values for scene-variation parameters is determined. The first set of values is automatically determined for generating a synthetic data scene frameset. The synthetic data scene frameset is generated based on the first set of values. The synthetic data scene frameset comprises at least a first frame in the frameset comprising the synthetic data scene updated based on a value for a scene-variation parameter. The synthetic data scene frameset is stored.

Distributed computing system with a synthetic data as a service frameset assembly engine

Data representing a scene is received. The scene includes labeled elements such as walls, a floor, a ceiling, and objects placed at various locations in the scene. The original received scene may be modified in different ways to create new scenes that are based on the original scene. These modifications include adding clutter to the scene, moving one or more elements of the scene, swapping one or more elements of the scene with different labeled elements, changing the size, color, or materials associated with one or more of the elements of the scene, and changing the lighting used in the scene. Each new scene may be used to generate labeled training data for a classifier by placing a virtual sensor (e.g., a camera) in the new scene, and generating sensor output data for the virtual sensor based on its placement in the new scene.

Generating training data for machine learning classifier

Various embodiments, methods and systems for implementing a distributed computing system asset assembly engine are provided. Initially a first source asset is received from a first distributed Synthetic Data as a Service (SDaaS) upload interface. A second source asset is received from a second a distributed SDaaS upload interface. The first source asset and the second source asset are ingested. Ingesting a source asset comprises automatically computing values for asset-variation parameters of the source asset, where the asset-variation parameters are programmable for machine-learning. A first synthetic data asset comprising a first set of values for the asset-variation parameters is generated. A second synthetic data asset comprising a second set of values for the asset-variation parameters is generated. The first synthetic data asset and the second synthetic data asset in a synthetic data asset are stored.

Distributed computing system with a synthetic data as a service asset assembly engine

Systems and methods are disclosed for permitting the use of a natural language expression to specify object (or asset) locations in a virtual three-dimensional (3D) environment. By rapidly identifying and solving constraints for 3D object placement and orientation, consumers of synthetics services may more efficiently generate experiments for use in development of artificial intelligence (AI) algorithms and sensor platforms. Parsing descriptive location specifications, sampling the volumetric space, and solving pose constraints for location and orientation, can produce large numbers of designated coordinates for object locations in virtual environments with reduced demands on user involvement. Converting from location designations that are natural to humans, such as 'standing on the floor one meter from a wall, facing the center of the room' to a six-dimensional (6D) pose specification (including 3-D location and orientation) can alleviate the need for a manual drag/drop/reorient procedure for placement of objects in a synthetic environment.

Growing community of inventors

Seattle, WA, United States of America

Michael Ebstyne