Magic Leap, Inc.

Meta Platforms Technologies, LLC

Facebook Technologies, LLC

Anush Mohan

Prateek Singhal

Ashwin Swaminathan

Ali Shahrokni

Xuan Zhao

Daniel Olshansky

Rafael Domingos Torres

Joel David Holder

Samuel A Miller

Robert Blake Taylor

Gioacchino Noris

Martin Georg Zahnert

Eran Guendelman

Miguel Andres Granados Velasquez

Jehangir Tajik

Christian Ivan Robert Moore

Lomesh Agarwal

Dominik Michael Kasper

Siddharth Choudhary

Joao Antonio Pereira Faro

Jeremy Dwayne Miranda

Chengyuan Yan

Divya Sharma

Ke Huo

Amrutha Hakkare Arunachala

Chengyuan Lin

Hiral Honar Barot

Michal Hlavac

Michael Harold Liebenow

Nicholas Atkinson Kramer

Benjamin Langmann

Sergiu Sima

Keng-Sheng Lin

Guy Rosenthal

Christopher Richard Tanner

Adi Perry

Paul Timothy Furgale

Kenneth William Tossell

Gilboa Sheveki

Johannes Joachim Schmid

Yuichi Taguchi

Shir Lene Lim

Wei Lyu

Matthew Banks

Andrea Alejandra Cohen

Sony Nguyen

Hongbing Hu

Hirai Honar Barot

An artificial reality environment (XRE) schema is defined that supports controlling interactions between various artificial reality actors. The XRE schema includes a set of definitions for an XRE, independent of type of artificial reality device. The definitions in the XRE schema can include standards for both interfaces and data objects. The XRE schema can define XR elements in terms of entities and components of a space, organized according to a hierarchy. Each entity can represent a real or virtual object or space, within the XRE, defined by a name and a collection of one or more components. Each component (as part of an entity) can define aspects and expose information about the entity. The XRE schema can specify structures that that allow actors (e.g., producers, instantiators, and consumers) to define and perform actions in relation to XRE elements.

Artificial reality environment control through an artificial reality environment schema

In particular embodiments, a computing system may initiate a scene capture process to capture a scene. The scene may include one or more of planes or objects. The system may send a first set of instructions to outline one or more planes of the scene. The system may cast a first set of rays to outline the one or more planes. The system may create the one or more planes based on the first set of rays. The system may send a second set of instructions to outline one or more objects of the scene. The system may cast a second set of rays to outline the one or more objects. The system may create the one or more objects based on the second set of rays. The system may generate a scene model of the scene based on the one or more planes and the one or more objects.

Scene capture via artificial reality systems

Implementations described herein can share scene data between multiple artificial reality (XR) devices, such as head-mounted displays (HMDs). A first XR device can retrieve stored spatial anchors associated with a scene, and align a coordinate frame with coordinate frames for a second XR device using the spatial anchors. The second XR device can have identified physical objects in the scene with corresponding object types defined in a scene lexicon (e.g., with a camera on the second XR device capturing images of physical objects and performing object recognition or through manual tagging of objects with object types), and store the object types and locations of the physical objects as scene data. The first XR device can obtain the scene data based on associations between the scene data and the spatial anchors and render virtual objects using the scene data, without having to rescan the scene for the physical objects.

Shared scene co-location for artificial reality devices

In particular embodiments, a computing system may initiate a scene alignment process to align a previous map of a scene with a current map of the scene. The system may send instructions to a user wearing an artificial-reality system to select a set of entities in the scene. The system may receive a selection of the set of entities in the scene. The system may determine a particular point in the scene based on an intersection of selected set of entities. The system may align the previous map with the current map based on the particular point in the scene. The system may load a scene model associated with the previous map into the current map.

Localization failure handling on artificial reality systems

To determine the head pose of a user, a head-mounted display system having an imaging device can obtain a current image of a real-world environment, with points corresponding to salient points which will be used to determine the head pose. The salient points are patch-based and include: a first salient point being projected onto the current image from a previous image, and with a second salient point included in the current image being extracted from the current image. Each salient point is subsequently matched with real-world points based on descriptor-based map information indicating locations of salient points in the real-world environment. The orientation of the imaging devices is determined based on the matching and based on the relative positions of the salient points in the view captured in the current image. The orientation may be used to extrapolate the head pose of the wearer of the head-mounted display system.

Enhanced pose determination for display device

An apparatus configured to be worn on a head of a user, includes: a screen configured to present graphics to the user; a camera system configured to view an environment in which the user is located; and a processing unit configured to determine a map based at least in part on output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; wherein the processing unit of the apparatus is also configured to obtain a metric indicating a likelihood of success to localize the user using the map, and wherein the processing unit is configured to obtain the metric by computing the metric or by receiving the metric.

Methods and apparatuses for determining and/or evaluating localizing maps of image display devices

An artificial reality environment (XRE) schema is defined that supports anel-controlling interactions between various artificial reality actors. The XRE schema includes a set of definitions for an XRE, independent of type of artificial reality device. The definitions in the XRE schema can include standards for both interfaces and data objects. The XRE schema can define XR elements in terms of entities and components of a space, organized according to a hierarchy. Each entity can represent a real or virtual object or space, within the XRE, defined by a name and a collection of one or more components. Each component (as part of an entity) can define aspects and expose information about the entity. The XRE schema can specify structures that that allow actors (e.g., producers, instantiators, and consumers) to define and perform actions in relation to XRE elements.

An apparatus configured to be head-worn by a user, includes: a screen configured to present graphics for the user; a camera system configured to view an environment in which the user is located; and a processing unit coupled to the camera system, the processing unit configured to: obtain locations of features for an image of the environment, wherein the locations of the features are identified by a neural network; determine a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image; and perform a corner detection using a corner detection algorithm to identify a corner in the region of interest.

Methods and apparatuses for corner detection using neural network and corner detector

A cross reality system enables any of multiple devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. The cross reality system may include a cloud-based localization service that responds to requests from devices to localize with respect to a stored map. The request may include one or more sets of feature descriptors extracted from an image of the physical world around the device. Those features may be posed relative to a coordinate frame used by the local device. The localization service may identify one or more stored maps with a matching set of features. Based on a transformation required to align the features from the device with the matching set of features, the localization service may compute and return to the device a transformation to relate its local coordinate frame to a coordinate frame of the stored map.

Cross reality system with localization service

A cross reality system that provides an immersive user experience by storing persistent spatial information about the physical world that one or multiple user devices can access to determine position within the physical world and that applications can access to specify the position of virtual objects within the physical world. Persistent spatial information enables users to have a shared virtual, as well as physical, experience when interacting with the cross reality system. Further, persistent spatial information may be used in maps of the physical world, enabling one or multiple devices to access and localize into previously stored maps, reducing the need to map a physical space before using the cross reality system in it. Persistent spatial information may be stored as persistent coordinate frames, which may include a transformation relative to a reference orientation and information derived from images in a location corresponding to the persistent coordinate frame.

Cross reality system

Systems and methods for providing a virtual space for multiple devices can include a first device having at least one sensor configured to acquire a spatial information of a physical space of the first device. The first device may include at least one processor configured to establish, according to the acquired spatial information, a virtual space corresponding to the physical space, that is accessible by a user of the first device via the first device. The at least one processor may further be configured to register a second device within the physical space, to allow a user of the second device to access the virtual space via the second device.

Systems and methods for maintaining virtual spaces

Growing community of inventors

Mountain View, CA, United States of America

Anush Mohan