Magic Leap, Inc.

Shiyu Dong

Prateek Singhal

Siddharth Choudhary

Khushi Gupta

Siddharth Mahendran

Divya Ramnath

Arumugam Kalai Kannan

Nitesh Sekhar

Manushree Gangwar

David Geoffrey Molyneaux

Frank Thomas Steinbruecker

Siddarth Mahendran

This document describes scene understanding for cross reality systems using occupancy grids. In one aspect, a method includes recognizing one or more objects in a model of a physical environment generated using images of the physical environment. For each object, a bounding box is fit around the object. An occupancy grid that includes a multiple cells is generated within the bounding box around the object. A value is assigned to each cell of the occupancy grid based on whether the cell includes a portion of the object. An object representation that includes information describing the occupancy grid for the object is generated. The object representations are sent to one or more devices.

Scene understanding using occupancy grids

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for scalable three-dimensional (3-D) object recognition in a cross reality system. One of the methods includes maintaining object data specifying objects that have been recognized in a scene. A stream of input images of the scene is received, including a stream of color images and a stream of depth images. A color image is provided as input to an object recognition system. A recognition output that identifies a respective object mask for each object in the color image is received. A synchronization system determines a corresponding depth image for the color image. A 3-D bounding box generation system determines a respective 3-D bounding box for each object that has been recognized in the color image. Data specifying one or more 3-D bounding boxes is received as output from the 3-D bounding box generation system.

Scalable three-dimensional object recognition in a cross reality system

A method to reconstruct an environment with depth information is provided. The method makes available to a wide variety of XR applications fresh and accurate 3D reconstruction data of environments with low processing time and low usage of computational resources and storage spaces. The method includes capturing depth information about an environment upon a change within a field-of-view of a user caused by, for example, motions of headpose, user location, and/or objects, and updating a representation of the environment based on depths associated with different confidence levels by the captured depth information. The method may include updating a geometry of the environment based on depths associated with confidence levels higher than a threshold. The method may include, upon a change of the 3D environment, inferring the removal of surfaces in the environment based on depths associated with confidence levels lower than the threshold.

Fast 3D reconstruction with depth information

Growing community of inventors

Santa Clara, CA, United States of America

Shiyu Dong