Meta Platforms Technologies, LLC

Disney Enterprises, Inc.

Facebook Technologies, LLC

Facebook, Inc.

Eidgenossische Technische Hochschule Zurich

Alessia Marra

Gioacchino Noris

Panya Inversin

Alexander Sorkine Hornung

Sarthak Ray

Robert Walker Sumner

Jennifer Lynn Spurlock

Mattia Ryffel

William Arthur Hugh Steptoe

Jeffrey Witthuhn

Björn Wanbo

Derrick Readinger

Sheng Shen

Fangwei Lee

Maurizio Nitti

Jonathan Mallinson

Marcus Tanner

Manuel Ricardo Freire Santos

Aleksejs Anpilogovs

Larissa Heike Laich

Connor Treacy

Javier Alejandro Sierra Santos

Federico Ruiz

Jonathan Richard Mayoh

Jeffrey Wai-King Lock

Michael James LeBeau

Stephane Magnenat

James Allan Booth

Adrian Brian Ratter

Fabio Zünd

Dat Tien Ngo

Yugeng He

Amir Mesguich Havilio

Kenny Mitchell

Matthew James Alderman

Fabian Langguth

Christopher Richard Tanner

Michael James Lebeau

Jeng-Weei Lin

Nicholas Wallen

Gerhard Roethlin

David Frederick Geisert

Michelle Pujals

In one embodiment, a method includes capturing, by a first VR display device, one or more frames of a shared real-world environment. The VR display device identifies one or more anchor points within the shared real-world environment from the one or more frames. The first VR display device receives localization information with respect to a second VR display device in the shared real-world environment and determines a pose of the first VR display device with respect to the second VR display device based on the localization information. A first output image is rendered for one or more displays of the first VR display device. The rendered image may comprise a proximity warning with respect to the second VR display device based on determining the pose of the first VR display device with respect to the second VR display device is within a threshold distance.

Spatial anchor sharing for multiple virtual reality systems in shared real-world environments

A method includes detecting an object of interest in a real environment and depth information of the object; determining one or more anchor locations in a three-dimensional space that correspond to a position of the object in the three-dimensional space; and generating a virtual surface anchored in the three-dimensional space. The method may further determine a pose of a camera when an image is captured and determine a region in the image that corresponds to the virtual surface. The method may further determine a first viewpoint of a first eye of the user; render a first output image based on (1) the first viewpoint relative to the virtual surface and (2) the image region corresponding to the virtual surface; and display the first output image on a first display of the computing device, the first display being configured to be viewed by the first eye of the user.

Systems and methods for generating stabilized images of a real environment in artificial reality

A method includes accessing an image of a physical environment of a user, the image depicting a physical input device and a physical hand of the user, determining that a contrast between the physical input device and the physical hand depicted in the image is lower than a predetermined threshold, modifying the image to increase the contrast, determining a pose of the physical input device, generating a three-dimensional model representing the physical hand of the user, generating an image mask by projecting the three-dimensional model onto an image plane, generating, a cropped image depicting at least the physical hand of the user in the image, rendering, based on the perspective of the user and the pose of the physical input device, a virtual input device to represent the physical input device, and displaying the cropped image depicting at least the physical hand over the rendered virtual input device.

Hand presence over keyboard inclusiveness

Aspects of the present disclosure are directed to creating and administering artificial reality collaborative working environments and providing interaction modes for them. An XR work system can provide and control such artificial reality collaborative working environments to enable, for example, A) links between real-world surfaces and XR surfaces; B) links between multiple real-world areas to XR areas with dedicated functionality; C) maintaining access, while inside the artificial reality working environment, to real-world work tools such as the user's computer screen and keyboard; D) various hand and controller modes for different interaction and collaboration modalities; E) use-based, multi-desk collaborative room configurations; and F) context-based auto population of users and content items into the artificial reality working environment.

Artificial reality collaborative working environments

In one embodiment, a method includes a computer system accessing an image of a physical environment of a user, the image being associated with a perspective of the user and depicting a physical input device and a physical hand of the user, determining a pose of the physical input device, generating a three-dimensional model representing the physical hand of the user, generating an image mask by projecting the three-dimensional model onto an image plane associated with the perspective of the user, generating, by applying the image mask to the image, a cropped image depicting at least the physical hand of the user in the image, rendering, based on the perspective of the user and the pose of the physical input device, a virtual input device to represent the physical input device, and displaying the cropped image depicting at least the physical hand over the rendered virtual input device.

A method includes a computing system tracking motions performed by a hand of a user, determining one or more anchor locations in a three-dimensional space, and generating a virtual surface anchored in the three-dimensional space. An image of a real environment is captured using a camera worn by the user, and a pose of the camera when the image is captured is determined. The computing system determines a first viewpoint of a first eye of the user and a region in the image that, as viewed from the camera, corresponds to the virtual surface. The computing system renders an output image based on (1) the first viewpoint relative to the virtual surface and (2) the image region corresponding to the virtual surface, and displays the output image on a first display of the device, the first display being configured to be viewed by the first eye of the user.

In one embodiment, a method includes displaying, for one or more displays of a virtual VR device, a first output image comprising a passthrough view of a real-world environment. The method includes identifying, using one or more images captured by one or more cameras of the VR display device, a real-world object in the real-world environment. The method includes receiving a user input indicating a first dimension corresponding to the real-world object. The method includes automatically determining, based on the first dimension, a second and third dimension corresponding to the real-world object. The method includes rendering, for the one or more displays of the VR display device, a second output image of a VR environment. The VR environment includes a MR object that corresponds to the real-world object. The MR object is defined by the determined first, second, and third dimensions.

Mixed reality objects in virtual reality environments

In one embodiment, a method includes accessing a plurality of points, wherein each point (1) corresponds to a spatial location associated with an observed feature of a physical environment and (2) is associated with a patch representing the observed feature, determining a density associated with each of the plurality of points based on the spatial locations of the plurality of points, scaling the patch associated with each of the plurality of points based on the density associated with the point, and reconstructing a scene of the physical environment based on at least the scaled patches.

Scene reconstruction from sparse data

An artificial reality system includes a head mounted display (HMD) and a physical overlay engine that generates overlay image data, referred to herein as a physical overlay image, corresponding to the physical objects in a three-dimensional (3D) environment. In response to an activation condition, a rendering engine of the artificial reality system renders the overlay image data to overlay artificial reality content for display on the HMD, thereby apprising a user of the HMD of their position with respect to the physical objects in the 3D environment.

Feed-forward collision avoidance for artificial reality environments

In one embodiment, a method includes displaying a first virtual content to a first user in a virtual area, the virtual area comprising one or more second virtual content, inferring an intent of the first user to interact with the first virtual content based on one or more of first user actions or contextual information, and adjusting one or more configurations associated with one or more of the second virtual content based on the inferring of the intent of the first user to interact with the first virtual content.

Virtual spaces, mixed reality spaces, and combined mixed reality spaces for improved interaction and collaboration

In one embodiment, a method includes receiving, from the first user, a request to create a joint virtual space to use with one or more second users, determining a first area in a first room associated with the first user based at least in part on space limitations associated with the first room and locations of one or more items in the first room, retrieving information associated with one or more second rooms for each of the second users, creating, based on first area of the first room and the information associated with each of the second rooms, the joint virtual space, and providing access to the joint virtual space to the first user and each of the one or more second users.

Growing community of inventors

Zurich, Switzerland

Alessia Marra