Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Renzo De Nardi

Richard Andrew Newcombe

Nicholas Daniel Trail

Zhaoming Zhu

Evan Gander

Steven John Lovegrove

Jason Victor Tsai

David Maurice Moore

Xinqiao Liu

Yuheng Ren

Raymond King

Brett D Christie

Robert Y Wang

Kieran Connor Kelly

Julian Straub

Thomas John Whelan

Shobhit Verma

Simon Gareth Green

Hauke Malte Strasdat

Peter Freese

Beipeng Mu

Daniel Miller

Evan Paul Gander

An Artificial Reality (AR) device captures an image of a physical environment surrounding a user wearing the AR device. A three-dimensional map corresponding to the physical environment is accessed. Then, a pose of the AR device relative to the three-dimensional map is determined based on first features of physical objects captured in the image and second features of object representations in the three-dimensional map. The device determines, using an eye tracker, a gaze of an eye of the user. Based on the gaze and the pose, a region of interest is computed in the three-dimensional map. Representations of physical devices in the region of interest of the three-dimensional map are identified. The device determines an intent of the user to interact with a physical device. A command is issued to the physical device based on the determined intent.

Method and a system for interacting with physical devices via an artificial-reality device

A system may include a wearable apparatus dimensioned to be worn by a user about an axial region of the user's body such that, when the wearable apparatus is worn by the user, the user's field of view into a local environment is substantially free of a view of the wearable apparatus. The system may also include a machine-perception subsystem that is coupled to the wearable apparatus and that gathers information about the local environment by observing the local environment. Additionally, the system may include an experience-analysis subsystem that infers, based on the information about the local environment and information about the user, contextual information about an experience of the user in the local environment. Furthermore, the system may include a non-visual communication subsystem that outputs the contextual information about the experience of the user. Various other apparatuses, systems, and methods are also disclosed.

Apparatus, system, and method for machine perception

A head-mounted display system may include a wearable frame securable to a user's head. The head-mounted display system may also include a mapping subsystem that maps a local environment of the user when the wearable frame is secured to the user's head. Additionally, the head-mounted display system may include a varifocal display apparatus mounted to the wearable frame and configured to direct computer-generated images toward the user's eyes at a variable focal length. Various other apparatuses, systems, and methods are also disclosed.

Apparatus, system, and method for improving digital head-mounted displays

A calibration system is described that includes a calibration target having one or more calibration objects, a head mounted display (HMD), and a rendering engine configured to render, based on an estimated pose of the HMD, respective virtual calibration objects in a viewing area of the HMD at respective locations corresponding to the one or more calibration objects visible in the viewing area, wherein the HMD is configured to output the virtual calibration targets. The calibration system further includes an image capture system configured to capture image data comprising images of the viewing area of the HMD, and a calibration engine configured to compute, from the image data, difference data comprising respective distances between the one or more calibration objects and the respective, corresponding virtual calibration objects.

End-to-end artificial reality calibration testing

A depth camera assembly (DCA) captures data describing depth information in a local area. The DCA includes an array detector, a controller, and an illumination source. The array detector includes a detector that is overlaid with a lens array. The detector includes a plurality of pixels, the plurality of pixels are divided into a plurality of different pixel groups. The lens array includes a plurality of lens stacks and each lens stack overlays a different pixel group. The array detector captures one or more composite images of the local area illuminated with the light from the illumination source. The controller determines depth information for objects in the local area using the one or more composite images.

Array detector for depth mapping

A depth measurement assembly (DMA) includes a structured light emitter, an augmented camera, and a controller. The structured light emitter projects structured light into a local area under instructions from the controller. The augmented camera generates image data of an object illuminated with the structured light pattern projected by the structured light emitter in accordance with camera instructions generated by the controller. The augmented camera includes a high speed computation tracking sensor that comprises a plurality of augmented photodetectors. Each augmented photodetector converts light to data and stores the data in its own memory unit. The controller receives the image data and determines depth information of the object in the local area based in part on the image data. The depth measurement unit can be incorporated into a head-mounted display (HMD).

High speed computational tracking sensor

A system includes an eyewear device configured to present content to a user. A processor is communicatively coupled to the eyewear device. A bracelet device is communicatively coupled to the processor, and includes at least one sensor configured to determine a position signal in response to movement of a user's hand. A depth camera assembly is communicatively coupled to the processor, and configured to emit a waveform into an environment of the user, and capture a plurality of optical signals from the waveform reflected off of at least one object in the environment. The processor is configured to determine a position of the user's hand in relation to the environment based in part on the position signal and the plurality of optical signals.

Distributed artificial reality system with contextualized hand tracking

The disclosed systems for positioning head-mounted displays may include a first track along which a display apparatus of a head-mounted display is configured to move in a first direction and a second track along which the display apparatus is configured to move in a second, different direction. A first electromechanical actuator may move the display apparatus along the first track, and a second electromechanical actuator may move the display apparatus along the second track. Various other methods and systems are also disclosed.

Systems and methods for positioning a head-mounted display

The disclosed computer-implemented method may include comprising identifying, within a real-world environment, a position of a user relative to a safety boundary. The position of the user is identified by a head-mounted display system comprising a display device. The display device is configured to at least partially obscure visibility of the real-world environment to the user. The method may further include selecting, based on the position of the user, at least a portion of a model of the real-world environment, rendering the portion of the model of the real-world environment, and displaying the rendered portion of the model of the real-world environment via the display device as a notification of the position of the user relative to the safety boundary. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for providing a mobile artificial reality user with environmental awareness

A system includes at least two base stations that emit light beams to illuminate an area for positional tracking objects in the area. A base station emits at least two light beams that rotate around a rotation axis at a rotational speed unique to the base station. Responsive to being illuminated by the light beams emitted by the at least two base stations, an object being tracked generates illumination data. The system determines which illumination data corresponds to one of multiple base stations by analyzing the illumination data over time. The system analyzes the illumination data corresponding to one base station to determine an orientation and/or position of the object relative to that base station.

Positional tracking using multiple base stations

Systems described herein use outside-in positional tracking. A base station emits one or more rotational light beams to illuminate a local area. The rotational light beams rotate around a rotation axis and are used for positional tracking one or more objects in the local area. The one or more rotational light beams retroreflect from the one or more objects in the local area. The one or more objects include retroreflectors that retroreflect light beams incident on them. The base station detects the retroreflected light beams. The base station generates illumination data in response to the detected light beams. A system analyzes the illumination to determine an orientation and/or a location of an object.

Positional tracking using retroreflectors

A depth camera assembly (DCA) that captures data describing depth information in a local area. The DCA includes an imaging device, a controller, and an illumination source. The illumination source includes a plurality of emitters on a single substrate. The imaging device captures one or more images of the local area illuminated with the light from the illumination source. The controller determines depth information for objects in the local area using the one or more images.

Multiple emitter illumination source for depth information determination

Systems described herein use outside-in positional tracking. A base station emits one or more rotational light beams to illuminate a local area. The rotational light beams rotate around a rotation axis and are used for positional tracking of one or more objects in the local area. A beam waist of a source light beam for generating the rotational light beams is positioned to be within a distance range from a center of rotation of the rotational light beams. An apparent frequency of the particular rotational light beam can be constant or substantially constant throughout a local area. Because the apparent frequency of the particular rotational light beam is constant or substantially constant throughout the local area, the frequency response of a detector in response to the illumination by the rotational light beam is within a limited range of values irrespective of the detector's location within the local area.

Growing community of inventors

Seattle, WA, United States of America

Renzo De Nardi