Microsoft Technology Licensing, LLC

University of North Carolina at Chapel Hill

Eidgenossische Technische Hochschule Zurich

Marc Andre Leon Pollefeys

David John McKinnon

Johannes Lutz Schonberger

Douglas Duane Berrett

Salim Sirtkaya

Yashar Bahman

Patrick Markus Misteli

Joshua Aidan Elsdon

Sing Bing Kang

Sudipta Narayan Sinha

Pablo Alejandro Speciale

Christoph Vogel

Raymond Kirk Price

Michael Bleyer

Yuri Pekelny

David Robert Gallup

Jan-Michael Frahm

Federica Bogo

Bugra Tekin

Ondrej Miksik

Lukas Gruber

Jan-Willem Buurlage

Juan Ignacio Nieto Couadeau

Timon Esli Knigge

Marcel Nicolas Geppert

A method for collecting telemetry data for updating a 3D map of an environment comprising carrying out a first relocalization event whereby: data from an observation at time t1 by a pose tracker is used to compute a first 3D map pose of the pose tracker, and a first local pose of the pose tracker at time t1 is received. A second relocalization event occurs where a second 3D map pose of the pose tracker is computed from data from an observation at time and a second local pose of the pose tracker at time t2 is received. A first relative pose between the first and second 3D map pose is computed. A second relative pose between the first and second local pose is computed. A residual being a difference between the first relative pose and the second relative pose is stored as input to a process for updating the 3D map.

Collecting telemetry data for 3D map updates

A method of tracking 3D position and orientation of an entity in a moving platform is described. The method comprises receiving data sensed by an inertial measurement unit mounted on the entity. Visual tracking data is also received, computed from images depicting the moving platform or the entity in the moving platform. The method computes the 3D position and orientation of the entity by estimating a plurality of states using the visual tracking data and the data sensed by the inertial measurement unit, where the states comprise both states of the moving platform and states of the entity.

Tracking in a moving platform

A method for authorizing access to one or more secured computer resources includes obfuscating a reference biometric vector into an obfuscated reference biometric vector using a similarity-preserving obfuscation. An authentication biometric vector is obfuscated into an obfuscated authentication biometric vector using the similarity-preserving obfuscation. A similarity of the obfuscated authentication biometric vector and the obfuscated reference biometric vector is tested. Based on the similarity being within an authentication threshold, access to the one or more secured computer resources is authorized.

Access authentication using obfuscated biometrics

Techniques for enabling a library of local maps to remain de-coupled from a global map are disclosed. An MR system is determined to be located on a platform that is currently moving or that has an ability to readily move. That platform's type is determined. Based on the determined type for the platform, a 3D boundary that approximates a shape for the platform's type is generated. The 3D boundary is imposed on the platform. Scanning data for the platform is acquired. The bounds for that scanning data is at least initially limited to that of the 3D boundary. The scanning data is used to build or supplement a library of local maps. The library is representative of the platform. That library is prevented from being coupled to a global map.

Enabling a local mixed reality map to remain de-coupled from a global mixed reality map

A 3D map comprising sensor data items depicting the environment is updated, each sensor data item having one or more associated variables such as a pose of a capture device or a position of a landmark. A graph is calculated from sensor data items. The graph comprises nodes and edges, a node representing at least one variable in the received sensor data items and an edge representing relationships between variables. The graph is partitioned into a plurality of subgraphs so as to reduce a number of variables that are shared between subgraphs. Each of the plurality of subgraphs is allocated to a respective worker node. At each worker node, updated values of the variables are computed. The process updates values of variables which are shared between subgraphs to a common value using a consensus process. The 3D map of the environment is updated according to the updated values of the variables.

Updating a 3D map of an environment

A method for video calling comprises, at a server computing system, receiving a plurality of segmented participant video streams from a plurality of client computing devices, each segmented participant video stream depicting a different human participant participating in a video call. One or more priority parameters for each of the plurality of human participants are recognized. One or more human participants are ranked based on a cumulative participant priority for each of the plurality of human participants. The plurality of segmented participant video streams are composited into a virtual conference view that displays each of the ranked one or more human participants at a virtual position based on their cumulative participant priority, such that human participants having higher cumulative participant priorities are displayed more prominently than human participants having lower cumulative participant priorities. The virtual conference view is sent to the plurality of client computing devices.

Virtual conference view for video calling

A three-dimensional (3D) map inconsistency detection machine includes an input transformation layer connected to a neural network. The input transformation layer is configured to 1) receive a test 3D map including 3D map data modeling a physical entity, 2) transform the 3D map data into a set of 2D images collectively corresponding to volumes of view frustums of a plurality of virtual camera views of the physical entity modeled by the test 3D map, and 3) output the set of 2D images to the neural network. The neural network is configured to output an inconsistency value indicating a degree to which the test 3D map includes inconsistencies based on analysis of the set of 2D images collectively corresponding to the volumes of the view frustums of the plurality of virtual camera views.

Three-dimensional map inconsistency detection using neural network

To obtain a relative localization between a plurality of mobile devices, a first mobile device observes a second mobile device within a field of view of the first mobile device's camera at time t, determines a first position of the first mobile device at t, and receives from the second mobile device a second position of the second mobile device at t. The first mobile device determines information about the first mobile device's orientation with respect to the second mobile device at tbased at least in part on the first position and the observation of the second mobile device. The first mobile device identifies two constraints that relate the mobile devices' coordinate systems based at least in part on the second position and the orientation information. The first mobile device's pose relative to the second mobile device may be calculated once at least six constraints are accumulated.

Relative spatial localization of mobile devices

A method for image-based localization includes, at a camera device, capturing a plurality of images of a real-world environment. A first set of image features are detected in a first image of the plurality of images. Before additional sets of image features are detected in other images of the plurality, the first set of image features is transmitted to a remote device configured to estimate a pose of the camera device based on image features detected in the plurality of images. As the additional sets of image features are detected in the other images of the plurality, the additional sets of image features are transmitted to the remote device. An estimated pose of the camera device is received from the remote device.

Image-based localization

A data processing system is provided that includes a processor having associated memory, the processor being configured to execute instructions using portions of the memory to cause the processor to, at classification time, receive an input image frame from an image source. The input image frame includes an articulated object and a target object. The processor is further caused to process the input image frame using a trained neural network configured to, for each input cell of a plurality of input cells in the input image frame predict a three-dimensional articulated object pose of the articulated object and a three-dimensional target object pose of the target object relative to the input cell. The processor is further caused to output the three-dimensional articulated object pose and the three-dimensional target object pose from the neural network.

Predicting three-dimensional articulated and target object pose

Computing devices and methods for estimating a pose of a user computing device are provided. In one example a 3D map comprising a plurality of 3D points representing a physical environment is obtained. Each 3D point is transformed into a 3D line that passes through the point to generate a 3D line cloud. A query image of the environment captured by a user computing device is received, the query image comprising query features that correspond to the environment. Using the 3D line cloud and the query features, a pose of the user computing device with respect to the environment is estimated.

Device pose estimation using 3D line clouds

Systems are provided for performing surface reconstruction with reduced power consumption. A surface mesh of an environment is generated, where the surface mesh is generated from multiple depth maps that are obtained of the environment. After the surface mesh is generated, a change detection image of that environment is captured while refraining from obtaining a new depth map of the environment. The change detection image is compared to the surface mesh. If a difference between the change detection image and the surface mesh is detected and if that difference satisfies a pre-determined difference threshold, then a new depth map of the environment is obtained. The surface mesh is then updated using the new depth map.

Growing community of inventors

Zurich, Switzerland

Marc Andre Leon Pollefeys