Microsoft Technology Licensing, LLC

Yuri Pekelny

Michael Bleyer

Raymond Kirk Price

Pedro Urbina Escos

Emanuel Shalev

Michael Ebstyne

Alex A Kipman

Jonathan Chi Hang Chan

Ali Osman Ulusoy

Christopher Douglas Edmonds

Szymon Piotr Stachniak

Andrey Konin

Michelle Brook

Mark Flick

Di Wang

Marc Andre Leon Pollefeys

Oliver Michael Strunk

Christopher Stephen Messer

Dimitrios Lymperopoulos

Muhammad Jabir Kapasi

Orion Wilson Elenzil

Vlad Stefan Burca

Matthew Alexander Joss

Rahul Sawhney

Salah Eddine Nouri

Michelle Lynn Brook

Ali O Ulusoy

A synthetic world interface may be used to model digital environments, sensors, and motions for the evaluation, development, and improvement of localization algorithms. A synthetic data cloud service with a library of sensor primitives, motion generators, and environments with procedural and game-like capabilities, facilitates engineering design for a manufactural solution that has localization capabilities. In some embodiments, a sensor platform simulator operates with a motion orchestrator, an environment orchestrator, an experiment generator, and an experiment runner to test various candidate hardware configurations and localization algorithms in a virtual environment, advantageously speeding development and reducing cost. Thus, examples disclosed herein may relate to virtual reality (VR) or mixed reality (MR) implementations.

Spatial localization design service

A head-mounted device (HMD) is configured to perform depth detection with a stereo camera pair comprising a first camera and a second camera, both of which are configured to detect/capture visible light and IR light. The fields of view for both of the cameras overlap to form an overlapping field of view. The HMD also includes an IR dot-pattern illuminator that is mounted on the HMD with the cameras and that is configured to emit an IR dot-pattern illumination. The IR dot-pattern illuminator emits a dot-pattern illumination that spans at least a part of the overlapping field of view. The IR dot-pattern illumination adds texture to objects in the environment and enables the HMD to determine depth for those objects, even if they have textureless/smooth surfaces.

Active stereo matching for depth applications

Improved techniques for re-localizing Internet-of-Things (IOT) devices are disclosed herein. Sensor data digitally representing one or more condition(s) monitored by an IOT device is received. In response, a sensor readings map is accessed, where this map is associated with the IOT device. The map also digitally represents the IOT device's environment and includes data representative of a location of the IOT device within the environment. The map also includes data representative of the conditions monitored by the IOT device. Additionally, the map is updated by attaching the sensor data to the map. In some cases, a coverage map can also be computed. Both the sensors readings map and the coverage map can be automatically updated in response to the IOT device being re-localized.

Mapping sensor data using a mixed-reality cloud

A synthetic world interface may be used to model digital environments, sensors, and motions for the evaluation, development, and improvement of computer vision and speech algorithms. A synthetic data cloud service with a library of sensor primitives, motion generators, and environments with procedural and game-like capabilities, facilitates engineering design for a manufactural solution that has computer vision and speech capabilities. In some embodiments, a sensor platform simulator operates with a motion orchestrator, an environment orchestrator, an experiment generator, and an experiment runner to test various candidate hardware configurations and computer vision and speech algorithms in a virtual environment, advantageously speeding development and reducing cost. Thus, examples disclosed herein may relate to virtual reality (VR) or mixed reality (MR) implementations.

Computer vision and speech algorithm design service

A computer implemented method includes obtaining a mesh representative of a scanned environment. Boundary edges of the mesh are identified. The boundary edges are grouped into ordered, connected three-dimensional closed non-intersecting polygons. A polygon is classified as a hole, and the area of the hole is determined.

Mesh hole area detection

The described implementations relate to reducing distance errors in depth maps. One example can detect blooming in a depth map. This example can identify a depth range affected by the blooming and take an action relating to the identified depth range to address the blooming.

Error reduction in depth maps

A method to process a contributing digital image of a subject in an image-processing computer. The contributing digital image is received in a depth-resolving machine configured to furnish a depth image based at least in part on the contributing digital image. The contributing digital image is also received in a classification machine previously trained to classify a pixel of the contributing digital image as liable to corrupt a depth value of a corresponding pixel of the depth image. A repair value is computed for the depth value of the corresponding pixel of the depth image, which is then corrected based on the repair value and returned to the calling process.

Pixel classification to reduce depth-estimation error

Optimizations are provided for reconstructing geometric surfaces for an environment that includes moving objects. Multiple depth maps for the environment are created, where some of the depth maps correspond to different perspectives of the environment. A motion state identifier is assigned to at least some pixels in at least some of the depth maps corresponding to moving objects in the environment. A composite 3D mesh is built using at least some of the multiple depth maps, by incorporating pixel information from the depth maps, while omitting pixel information identified by the motion state identifiers as being associated with moving objects.

Surface reconstruction for environments with moving objects

Disclosed herein are optimized techniques for controlling the exposure time or illumination intensity of a depth sensor. Invalid-depth pixels are identified within a first depth map of an environment. For each invalid-depth pixel, a corresponding image pixel is identified in a depth image that was used to generate the first depth map. Multiple brightness intensities are identified from the depth image. Each brightness intensity is categorized as corresponding to either an overexposed or underexposed image pixel. An increased exposure time or illumination intensity or, alternatively, a decreased exposure time or illumination intensity is then used to capture another depth image of the environment. After a second depth map is generated based on the new depth image, portion(s) of the second depth map are selectively merged with the first depth map by replacing the invalid-depth pixels of the first depth map with corresponding valid-depth pixels of the second depth map.

Optimized exposure control for improved depth mapping

An immersive feedback loop is disclosed for improving artificial intelligence (AI) applications used for virtual reality (VR) environments. Users may iteratively generate synthetic scene training data, train a neural network on the synthetic scene training data, generate synthetic scene evaluation data for an immersive VR experience, indicate additional training data needed to correct neural network errors indicated in the VR experience, and then generate and retrain on the additional training data, until the neural network reaches an acceptable performance level.

Immersive feedback loop for improving AI

Improved techniques for re-localizing Internet-of-Things (IOT) devices are disclosed herein. Sensor data digitally representing one or more condition(s) monitored by an IOT device is received. In response, a sensor readings map is accessed, where this map is associated with the IOT device. The map also digitally represents the IOT device's environment and includes data representative of a location of the IOT device within the environment. The map also includes data representative of the conditions monitored by the IOT device. Additionally, the map is updated by attaching the sensor data to the map. In some cases, a coverage map can also be computed. Both the sensors readings map and the coverage map can be automatically updated in response to the TOT device being re-localized.

A computer implemented method includes obtaining a first deep neural network (DNN) model trained on labeled real image data for a downstream vision task, obtaining a second DNN model trained on synthetic images created with random image parameter values for the downstream vision task, obtaining a third DNN model trained on the labeled real image data and the synthetic images for the downstream vision task, performing a forward pass execution of each model to generate a loss, backpropagating the loss to modify parameter values, and iterating the forward pass execution and backpropagating with images generated by the modified parameters to jointly train the models and optimize the parameters.

Growing community of inventors

Seattle, WA, United States of America

Yuri Pekelny