Magic Leap, Inc.

Google Inc.

Xiaolin Wei

Yifu Zhang

Jianyuan Min

David Geoffrey Molyneaux

Frank Thomas Steinbrücker

Zhongle Wu

Tilman Reinhardt

Aleksey Golovinskiy

Hui Lou

A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.

Viewpoint dependent brick selection for fast volumetric reconstruction

A method for refining poses includes receiving a plurality of poses and performing volumetric reconstruction for the plurality of poses to produce a global geometry. The method also includes refining each pose of the plurality of poses with respect to the global geometry to produce a plurality of refined poses.

Methods and systems for large-scale determination of RGBD camera poses

A method for updating camera poses includes receiving a set of captured depth maps associated with a scene and detecting a first shape and a second shape present in the scene. The method also includes, for each of the first shape and the second shape, creating a 3D mesh, creating a virtual camera associated with the 3D mesh, and rendering a depth map associated with the virtual camera. The method further includes identifying a subset of the captured depth maps such that each captured depth map in the subset contains at least a portion of either the first shape or the second shape. The method additionally includes updating the physical camera poses by jointly solving for the physical camera poses by optimizing an alignment between the first depth map, the second depth map, and the subset of the set of captured depth maps.

Methods and systems for detecting and combining structural features in 3D reconstruction

An augmented reality/mixed reality system that provides a more immersive user experience. That experience is provided with increased speed of update for occlusion data by using depth sensor data augmented with lower-level reconstruction data. When operating in real-time dynamic environments, changes in the physical world can be reflected quickly in the occlusion data. Occlusion rendering using live depth data augmented with lower-level 3D reconstruction data, such as a raycast point cloud, can greatly reduce the latency for visual occlusion processing. Generating occlusion data in this way may provide faster operation of an XR system using less computing resources and enabling the system to be packaged in a battery operated wearable device.

Method of occlusion rendering using raycast and live depth

A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications.

Caching and updating of dense 3D reconstruction data

A method of operating a computing system to generate a model of an environment represented by a mesh is provided. The method allows to update 3D meshes to client applications in real time with low latency to support on the fly environment changes. The method provides 3D meshes adaptive to different levels of simplification requested by various client applications. The method provides local update, for example, updating the mesh parts that are changed since last update. The method also provides 3D meshes with planarized surfaces to support robust physics simulations. The method includes segmenting a 3D mesh into mesh blocks. The method also includes performing a multi-stage simplification on selected mesh blocks. The multi-stage simplification includes a pre-simplification operation, a planarization operation, and a post-simplification operation.

Multi-stage block mesh simplification

A method of merging 3D meshes includes receiving a first mesh and a second mesh; performing spatial alignment to register the first mesh and the second mesh in a common world coordinate system; performing mesh clipping on the first mesh and the second mesh to remove redundant mesh vertices; performing geometry refinement around a clipping seam to close up mesh concatenation holes created by mesh clipping; and performing texture blending in regions adjacent the clipping seam to obtain a merged mesh.

Robust merge of 3D textured meshes

A method for forming a reconstructed 3D mesh includes receiving a set of captured depth maps associated with a scene, performing an initial camera pose alignment associated with the set of captured depth maps, and overlaying the set of captured depth maps in a reference frame. The method also includes detecting one or more shapes in the overlaid set of captured depth maps and updating the initial camera pose alignment to provide a shape-aware camera pose alignment. The method further includes performing shape-aware volumetric fusion and forming the reconstructed 3D mesh associated with the scene.

Growing community of inventors

Fremont, CA, United States of America

Xiaolin Wei