Meta Platforms, Inc.

Gian Diego Tipaldi

Gioacchino Noris

Alexander Sorkine Hornung

Alberto Garcia Garcia

Gregory Mayo Daly

Alexandru-Eugen Ichim

Mahdi Salmani Rahimi

Gaurav Chaurasia

Matthew James Alderman

Arthur Nieuwoudt

Jan Oberländer

Lingwen Gan

Paul Timothy Furgale

Mark David Strachan

In one embodiment, a computing system determines one or more depth measurements associated with a first physical object. The system captures an image including image data associated with the first physical object. The system identifies and associates a plurality of first pixels with a first representative depth value based on the one or more depth measurements. The system determines, for an output pixel of an output image, that (1) a portion of a virtual object is visible from a viewpoint and (2) the output pixel overlaps with a portion of the first physical object. The system determines that the portion of the first physical object is associated with the plurality of first pixels and renders the output image from the viewpoint. Occlusion at the output pixel is determined based on a comparison between the first representative depth value and a depth value associated with the portion of the virtual object.

Systems, methods, and media for generating visualization of physical environment in artificial reality

In one embodiment for generating passthrough, a system may receive an image and depth measurements of an environment and generate a corresponding 3D model. The system identifies, in the image, first pixels depicting a physical object and second pixels corresponding to a padded boundary around the first pixels. The system associates the first pixels with a first portion of the 3D model representing the physical object and a first representative depth value computed based on the depth measurements. The system associates the second pixels with a second portion of the 3D model representing a region around the physical object and a second representative depth value farther than the first representative depth value. The system renders an output image depicting a virtual object and the physical object. Occlusions between the virtual object and the physical object are determined using the first representative depth value and the second representative depth value.

In one embodiment for generating passthrough, a computing system may access images of an environment captured by cameras of a device worn by a user. The system may generate, based on the images, depth measurements of objects in the environment. The system may generate a mesh covering a field of view of the user and then update the mesh based on the depth measurements to represent a contour of the objects in the environment. The system may determine a first viewpoint of a first eye of the user and render a first output image based on the first viewpoint and the updated mesh. The system may then display the first output image on a first display of the device, the first display being configured to be viewed by the first eye of the user.

A method includes receiving an image of a real environment using a camera worn by a user, and determining a portion of the image that comprises an object of interest. Based on the portion of the image that comprises the object of interest, a surface representing the object of interest is generated. Depth measurements of the real environment corresponding to the portion of the image comprising the object of interest are received and used to determine a depth of the surface representing the object of interest. The surface is posed in a coordinate system corresponding to the real environment based on the depth of the surface and a visibility of a virtual object is determined relative to the object of interest by comparing a model of the virtual object with the surface. The output image is generated based on the determined visibility of the virtual object.

Generating and modifying representations of dynamic objects in an artificial reality environment

In one embodiment for generating passthrough, a computing system may compute, based on an image of a physical environment, depth measurements of at least one physical object. The system may generate a first model of the physical object using the depth measurements. The system may identify first pixels in the image that depict the physical object and associate them with a first representative depth value computed using the first model. The system may determine, for a pixel of an output image, that a portion of the first model and a portion of a second model of a virtual object are visible. The system may determine that the portion of the first model is associated with the plurality of first pixels and determine occlusion at the pixel based on a comparison between the first representative depth value and a depth value associated with the portion of the second model.

Growing community of inventors

Zurich, Switzerland

Gian Diego Tipaldi