Apple Inc.

Yan Cote

Bertrand Nepveu

Marc-Andre Chenier

Yves Millette

Anthony Ghannoum

Vincent Chapdelaine-Couture

Simon Fortin-Deschenes

Tomlinson M Holman

Simon Fortin-Deschênes

Marc-Olivier Lepage

Sandy J Carter

Irving Lustigman

A Head-Mounted Display system together with associated techniques for performing accurate and automatic inside-out positional, user body and environment tracking for virtual or mixed reality are disclosed. The system uses computer vision methods and data fusion from multiple sensors to achieve real-time tracking. High frame rate and low latency is achieved by performing part of the processing on the HMD itself.

Head-mounted display for virtual and mixed reality with inside-out positional, user body and environment tracking

The present disclosure relates to techniques for inserting imagery from a real environment into a virtual environment. While presenting (e.g., displaying) the virtual environment at an electronic device, a proximity of the electronic device to a physical object located in a real environment is detected. In response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance, imagery of the physical object is isolated from other imagery of the real environment. The isolated imagery of the physical object is inserted into the virtual environment at a location corresponding to the location of the physical object in the real environment. The imagery of the physical object has a first visibility value associated with the proximity of the electronic device to the physical object.

Inserting imagery from a real environment into a virtual environment

A Head-Mounted Display with camera sensors to perform chroma keying in a mixed reality context is presented. Low latency is achieved by embedding the processing in the HMD itself, specifically, format camera images, detect the selected color range and make a composite with the virtual content.

Low latency chroma keying embedded in a head-mounted display for mixed reality

A method includes obtaining a first plurality of image frames that are provided by an image sensor at a first output rate. The method includes determining a GPU temporal processing value characterizing the GPU processing of the first plurality of image frames. The GPU temporal processing value is a function of an amount of information associated with the first plurality of image frames. The method includes determining an operation value associated with the image sensor based on a function of the GPU temporal processing value. The operation value modifies the operation of the image sensor. The method includes changing, using a controller, the operation of the image sensor as a function of the operation value so that the image sensor provides a second plurality of image frames at a second output rate.

Controlling an image sensor based on GPU temporal processing values

In some implementations, a method includes: determining a complexity value for first image data associated with of a physical environment that corresponds to a first time period; determining an estimated composite setup time based on the complexity value for the first image data and virtual content for compositing with the first image data; in accordance with a determination that the estimated composite setup time exceeds the threshold time: forgoing rendering the virtual content from the perspective that corresponds to the camera pose of the device relative to the physical environment during the first time period; and compositing a previous render of the virtual content for a previous time period with the first image data to generate the graphical environment for the first time period.

Method and device for latency reduction of an image processing pipeline

In some implementations, a method of reducing latency associated with an image read-out operation is performed at a device including one or more processors, non-transitory memory, an image processing architecture, and an image capture device. The method includes: obtaining first image data corresponding to a physical environment; reading a first slice of the first image data into an input buffer; performing processing operations on the first slice of the first image data to obtain a first portion of second image data; reading a second slice of the first image data into the input buffer; performing the image processing operations on the second slice of the first image data to obtain a second portion of the second image data; and generating an image frame of the physical environment based at least in part on the first and second portions of the second image data.

Generating a representation of a scene includes detecting an indication to capture sensor data to generate a virtual representation of a scene in a physical environment at a first time, in response to the indication obtaining first sensor data from a first capture device at the first time, obtaining second sensor data from a second capture device at the first time, and combining the first sensor data and the second sensor data to generate the virtual representation of the scene.

Growing community of inventors

Santa Clara, CA, United States of America

Yan Cote