Meta Platforms Technologies, LLC

Facebook Technologies, LLC

Disney Enterprises, Inc.

Facebook, Inc.

Meta Platforms, Inc.

Eth Zurich (eidgenoessische Technische Hochschule Zurich)

Gioacchino Noris

Alexander Sorkine Hornung

Alessia Marra

Sarthak Ray

Panya Inversin

Alexandru-Eugen Ichim

James Allan Booth

Robert Walker Sumner

Gaurav Chaurasia

Jennifer Lynn Spurlock

William Arthur Hugh Steptoe

Jeffrey Witthuhn

Derrick Readinger

Sheng Shen

Fangwei Lee

Jonathan Mallinson

Marcus Tanner

Federico Ruiz

Connor Treacy

Larissa Heike Laich

Javier Alejandro Sierra Santos

Aleksejs Anpilogovs

Manuel Ricardo Freire Santos

Anush Mohan

Jonathan Richard Mayoh

Jeffrey Wai-King Lock

Michael James LeBeau

Maryann Simmons

Jan Oberländer

Gian Diego Tipaldi

Christopher Richard Tanner

Markus Gross

Stelian Coros

Michal Hlavac

Mahdi Salmani Rahimi

Brian Whited

Manuel Werlberger

Björn Wanbo

Matthew James Alderman

Jeng-Weei Lin

Alex Locher

Paul Timothy Furgale

Johannes Joachim Schmid

Daniel Sykora

Alberto Garcia Garcia

Markus H Gross

Amir Mesguich Havilio

Kenny Mitchell

Mattia Ryffel

Ariel Shamir

Maurizio Nitti

Arthur Nieuwoudt

Stephane Magnenat

Michael James Lebeau

David Frederick Geisert

Gerhard Roethlin

Eldad Yitzhak

Nicholas Wallen

Michelle Pujals

James Allen Booth

Hongbing Hu

Sony Nguyen

Andrea Alejandra Cohen

Matthew Banks

An artificial reality environment (XRE) schema is defined that supports controlling interactions between various artificial reality actors. The XRE schema includes a set of definitions for an XRE, independent of type of artificial reality device. The definitions in the XRE schema can include standards for both interfaces and data objects. The XRE schema can define XR elements in terms of entities and components of a space, organized according to a hierarchy. Each entity can represent a real or virtual object or space, within the XRE, defined by a name and a collection of one or more components. Each component (as part of an entity) can define aspects and expose information about the entity. The XRE schema can specify structures that that allow actors (e.g., producers, instantiators, and consumers) to define and perform actions in relation to XRE elements.

Artificial reality environment control through an artificial reality environment schema

In particular embodiments, a computing system may initiate a scene capture process to capture a scene. The scene may include one or more of planes or objects. The system may send a first set of instructions to outline one or more planes of the scene. The system may cast a first set of rays to outline the one or more planes. The system may create the one or more planes based on the first set of rays. The system may send a second set of instructions to outline one or more objects of the scene. The system may cast a second set of rays to outline the one or more objects. The system may create the one or more objects based on the second set of rays. The system may generate a scene model of the scene based on the one or more planes and the one or more objects.

Scene capture via artificial reality systems

In particular embodiments, a computing system may initiate a scene alignment process to align a previous map of a scene with a current map of the scene. The system may send instructions to a user wearing an artificial-reality system to select a set of entities in the scene. The system may receive a selection of the set of entities in the scene. The system may determine a particular point in the scene based on an intersection of selected set of entities. The system may align the previous map with the current map based on the particular point in the scene. The system may load a scene model associated with the previous map into the current map.

Localization failure handling on artificial reality systems

In one embodiment, a method includes capturing, by a first VR display device, one or more frames of a shared real-world environment. The VR display device identifies one or more anchor points within the shared real-world environment from the one or more frames. The first VR display device receives localization information with respect to a second VR display device in the shared real-world environment and determines a pose of the first VR display device with respect to the second VR display device based on the localization information. A first output image is rendered for one or more displays of the first VR display device. The rendered image may comprise a proximity warning with respect to the second VR display device based on determining the pose of the first VR display device with respect to the second VR display device is within a threshold distance.

Spatial anchor sharing for multiple virtual reality systems in shared real-world environments

In one embodiment, a computing system determines one or more depth measurements associated with a first physical object. The system captures an image including image data associated with the first physical object. The system identifies and associates a plurality of first pixels with a first representative depth value based on the one or more depth measurements. The system determines, for an output pixel of an output image, that (1) a portion of a virtual object is visible from a viewpoint and (2) the output pixel overlaps with a portion of the first physical object. The system determines that the portion of the first physical object is associated with the plurality of first pixels and renders the output image from the viewpoint. Occlusion at the output pixel is determined based on a comparison between the first representative depth value and a depth value associated with the portion of the virtual object.

Systems, methods, and media for generating visualization of physical environment in artificial reality

A method includes detecting an object of interest in a real environment and depth information of the object; determining one or more anchor locations in a three-dimensional space that correspond to a position of the object in the three-dimensional space; and generating a virtual surface anchored in the three-dimensional space. The method may further determine a pose of a camera when an image is captured and determine a region in the image that corresponds to the virtual surface. The method may further determine a first viewpoint of a first eye of the user; render a first output image based on (1) the first viewpoint relative to the virtual surface and (2) the image region corresponding to the virtual surface; and display the first output image on a first display of the computing device, the first display being configured to be viewed by the first eye of the user.

Systems and methods for generating stabilized images of a real environment in artificial reality

An artificial reality environment (XRE) schema is defined that supports anel-controlling interactions between various artificial reality actors. The XRE schema includes a set of definitions for an XRE, independent of type of artificial reality device. The definitions in the XRE schema can include standards for both interfaces and data objects. The XRE schema can define XR elements in terms of entities and components of a space, organized according to a hierarchy. Each entity can represent a real or virtual object or space, within the XRE, defined by a name and a collection of one or more components. Each component (as part of an entity) can define aspects and expose information about the entity. The XRE schema can specify structures that that allow actors (e.g., producers, instantiators, and consumers) to define and perform actions in relation to XRE elements.

A system generates a plurality of spatial points based on depth measurements of physical objects. The system determines, based on the plurality of spatial points, an occupancy score for each voxel within a plurality of voxels. The system identifies, based on a gaze of the user, a first set of occupied voxels that are in a field of view of the user and a second set of occupied voxels that are outside the field of view of the user. The system updates the occupancy scores of the first set of occupied voxels by temporally decaying one or more of the plurality of spatial points within the first set of occupied voxels. The system maintains the occupancy scores of the second set of occupied voxels. The system detects intrusions in a predefined subspace within a physical space based on the updated occupancy scores of the first set of occupied voxels.

Systems and methods for detecting objects within the boundary of a defined space while in artificial reality

Aspects of the present disclosure are directed to creating and administering artificial reality collaborative working environments and providing interaction modes for them. An XR work system can provide and control such artificial reality collaborative working environments to enable, for example, A) links between real-world surfaces and XR surfaces; B) links between multiple real-world areas to XR areas with dedicated functionality; C) maintaining access, while inside the artificial reality working environment, to real-world work tools such as the user's computer screen and keyboard; D) various hand and controller modes for different interaction and collaboration modalities; E) use-based, multi-desk collaborative room configurations; and F) context-based auto population of users and content items into the artificial reality working environment.

Artificial reality collaborative working environments

In one embodiment, a method includes segmenting a layout of a physical space surrounding a user into physical segments; generating, based on the physical segments, virtual paths for a virtual environment through which the user can navigate by traveling the physical segments; identifying, based on a current location of the user with respect to the physical space, a portion of the physical segments for which to enable an intrusion detection feature; detecting a physical object in the portion of the physical segments that corresponds to a particular virtual path of the virtual paths; and in response to the detecting, displaying a representation of the physical object in the particular virtual path.

Comfortable navigation in large-scale virtual worlds when constrained by small-scale physical environment

The disclosed artificial reality system can provide a user self representation in an artificial reality environment based on a self portion from an image of the user. The artificial reality system can generate the self representation by applying a machine learning model to classify the self portion of the image. The machine learning model can be trained to identify self portions in images based on a set of training images, with portions tagged as either depicting a user from a self-perspective or not. The artificial reality system can display the self portion as a self representation in the artificial reality environment by positioning them in the artificial reality environment relative to the user's perspective in the artificial reality environment. The artificial reality system can also identify movements of the user and can adjust the self representation to match the user's movement, providing more accurate self representations.

Self presence in artificial reality

A mixed reality (MR) simulation system includes a console and a head mounted device (HMD). The MR system captures stereoscopic images from a real-world environment using outward-facing stereoscopic cameras mounted to the HMD. The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. For each feature, the MR system generates one or more two-dimensional (2D) polylines. Then, the MR system triangulates between 2D polylines found in right side images and corresponding 2D polylines found in left side images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating a geometric reconstruction of the real-world environment. The MR system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.

Reconstruction of essential visual cues in mixed reality applications

Growing community of inventors

Zurich, Switzerland

Gioacchino Noris