Magic Leap, Inc.

Frank Thomas Steinbruecker

Ashwin Swaminathan

David Geoffrey Molyneaux

Daniel E Koppel

Lipu Zhou

Mohan Babu Kandra

Shiyu Dong

Vinayram Balakumar

Hui Ju

Konstantinos Zampogiannis

Pooja Piyush Mehta

A method of efficiently and accurately computing a pose of an image with respect to other image information. The image may be acquired with a camera on a portable device and the other information may be a map, such that the computation of pose localizes the device relative to the map. Such a technique may be applied in a cross reality system to enable devices to efficiently and accurately access previously persisted maps. Localizing with respect to a map may enable multiple cross reality devices to render virtual content at locations specified in relation to those maps, providing an enhanced experience for uses of the system. The method may be used in other devices and for other purposes, such as for navigation of autonomous vehicles.

Efficient localization based on multiple feature types

Methods and apparatus for providing a representation of an environment, for example, in an XR system, and any suitable computer vision and robotics applications. A representation of an environment may include one or more planar features. The representation of the environment may be provided by jointly optimizing plane parameters of the planar features and sensor poses that the planar features are observed at. The joint optimization may be based on a reduced matrix and a reduced residual vector in lieu of the Jacobian matrix and the original residual vector.

Computationally efficient method for computing a composite representation of a 3D environment

A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications. Between sessions of an XR device, blocks may be persisted on the device or in remote storage in one or more cache layers. The device may, upon starting a new session, selectively use the blocks from one or more layers of the cache.

Caching and updating of dense 3D reconstruction data

A method to reconstruct an environment with depth information is provided. The method makes available to a wide variety of XR applications fresh and accurate 3D reconstruction data of environments with low processing time and low usage of computational resources and storage spaces. The method includes capturing depth information about an environment upon a change within a field-of-view of a user caused by, for example, motions of headpose, user location, and/or objects, and updating a representation of the environment based on depths associated with different confidence levels by the captured depth information. The method may include updating a geometry of the environment based on depths associated with confidence levels higher than a threshold. The method may include, upon a change of the 3D environment, inferring the removal of surfaces in the environment based on depths associated with confidence levels lower than the threshold.

Fast 3D reconstruction with depth information

Growing community of inventors

Mountain View, CA, United States of America

Frank Thomas Steinbruecker