Apple Inc.

Tom Sengelaub

Hua Gao

Hao Qin

Anselm Grundhoefer

Mohamed Selim Ben Himane

Thomas G Salter

Arun Srivatsan Rangaprasad

Jeffrey S Norris

Paul Ewers

Scott G Wade

Jonathan P Ive

Alan C Dye

Stephen Olivier Lemay

Julian Jaede

Fletcher R Rothkopf

Cheng Kai Chen

Jeremy C Franklin

William A Sorrentino, Iii

Jason C Sauers

Geoffrey Grant Stahl

Arthur Yasheng Zhang

Wey-Jiun Lin

Ivan S Maric

Phil M Hobson

Graham Butler Myhre

Michael J Rockwell

Forrest C Wang

Yoonhoo Jo

Muhammad F Hossain

Ritu Shah

Jia Tao

Marinus Meursing

Yanghai Tsin

Kathrin Berkner Cieslicki

Bertrand Nepveu

Ryan J Dunn

Jan K Quijalvo

Julian Hoenig

Jason L Slupeiks

Peter Kaufmann

Rene Heideklang

Uma Arunachalam

Marvin Klinkhardt

Hesam Najafi

Tobias Eble

James W Vandyke

Branko Petljanski

Thibaut Weise

Harihar Narasimha-Iyer

Martin Haller

Jie Zhong

Chia-Yin Tsai

Methods and apparatus for flexible illumination that improve the performance and robustness of an imaging system are described. Multiple different lighting configurations for the imaging system are pre-generated. Each lighting configuration may specify one or more aspects of lighting. A lookup table may be generated via which each pose is associated with a respective lighting configuration. A user may put on, hold, or otherwise use the device. A process may be initiated in which different lighting configurations may be selected by the controller to capture images of the user's eye, periorbital region, or face at different poses and in different conditions for use by a biometric authentication or gaze tracking process.

Flexible illumination for imaging systems

In an unobtrusive visual axis enrollment process, a line of text or other content is displayed at a known vertical location and virtual depth that the user can then read. This line of text may be content that the user needs to read as part of the normal enrollment process. As the user reads the line of text, eye tracking cameras may capture images of the eye. This data may then be used to estimate a stimulus plane. The error between the estimated stimulus plane and the ground truth stimulus plane (the actual location of the line of text in virtual space) may then be used to estimate the kappa angle.

Visual axis enrollment

A personalized eye model is used to generate synthetic gaze features at ground-truth eye poses G. Corresponding synthetic gaze poses Gare estimated from the synthetic gaze features using an average eye model. A linear regression is applied between Gand Gto generate a gaze correction function. The gaze correction function represents differences between the synthetic gaze Gof the subject eye at the display and that of the average eye model Gat the display, but does not contain security- or privacy-sensitive information. Further, the personalized eye model cannot be recovered from the gaze correction function, and thus the gaze correction function can be stored unencrypted and available for use during a cold boot of a device prior to login. On a cold boot of the device, the gaze correction function may be accessed and used with an average eye model to improve gaze-based interactions.

Synthetic gaze enrollment

Systems and methods are disclosed to enable detection of excessive contact lens shift in a head-mounted display (HMD) device that implement gaze tracking for the user. In embodiments, during a user enrollment process to calibrate the gaze tracking system, the HMD device analyzes key frames of the eye to estimate a first center location of the eye based on detected eye features such as the pupil, the iris, or the limbus, and a second center location of the eye based on the shape of the detected cornea. Shift vectors are generated based on the two center locations for individual key frames. If the standard deviation of the vectors' magnitudes exceeds a specified threshold, excessive lens shift is detected. In response to detection of excessive contact lens shift, the HMD device may generate a notification to the user or activate compensation mechanisms.

Contact lens shift detection for head-mounted display devices

Head-mounted devices (HMDs) include display screens for display of video and/or images. A complex lens is positioned between the display screen and a user's eye to render images shown on the display screen for correct appearance to the user's eye. Changes in relative position or orientation between the display and the lens may cause unwanted changes in the appearance of the images viewed by the user. A process is disclosed to determine changes in relative positions and orientation of the display screen and lens. By determining changes in relative positions and orientation, recalibration of the system may be implemented to continually present images to the user's eye with the proper focus for a more satisfactory user experience.

Camera and display calibration

A head-mountable display device includes a housing defining a front opening and a rear opening, a display screen disposed in the front opening, a display assembly disposed in the rear opening, a first securement strap coupled to the housing, the first securement strap including a first electronic component, a second securement strap coupled to the housing, the second securement strap including a second electronic component, and a securement band extending between and coupled to the first securement strap and the second securement strap.

Head mountable display

Various implementations disclosed herein include devices, systems, and methods that provide detecting and decoding of visual markers that encode data. In some implementations, a visual marker is detected in an image of a physical environment. In some implementations, detecting the visual marker is based on a pre-determined portion of the visual marker. In some implementations, detecting the visual marker is based on a predefined shape. In some implementations, a visual marker interpretation scheme is determined for decoding the visual marker. In some implementations, the visual marker is decoded based on one or more colors being attributed a same value. In some implementations, the visual marker is decoded based on one or more other colors being attributed a different value.

Visual marker with user selectable appearance

Operating an outward-facing display device for low light conditions may include receiving information indicative of ambient lighting conditions for the display device's environment; in accordance with a determination that the ambient lighting conditions do not satisfy a brightness criterion, determining a set of pixels in the outward-facing display device corresponding to an area of interest in the environment; and driving the set of pixels at a particular brightness to improve lighting in the area of interest.

Electronic device with a display for low light conditions

Various implementations disclosed herein include devices, systems, and methods that assess calibration between eye tracking and other components of a head-mounted device (HMD) using another device, such as a mobile device. For example, an example process may include obtaining first sensor data captured by a first sensor of a first device, the first sensor data including a representation of a portion of a second device, obtaining second sensor data captured by a second sensor of the second device, detecting a position of the first sensor of the first device based on the second sensor data, and assessing a calibration between the portion of the second device and the sensor of the second device based on the first sensor data and the detected position of the first sensor.

Display calibration

Various implementations disclosed herein include devices, systems, and methods that provide color visual markers that include colored markings that encode data, where the colors of the colored markings are determined by scanning (e.g., detecting the visual marker using a sensor of an electronic device) the visual marker itself. In some implementations, a visual marker is detected in an image of a physical environment. In some implementations, the visual marker is detected in the image by detecting a predefined shape of a first portion of the visual marker in the image. Then, a color-interpretation scheme is determined for interpreting colored markings of the visual marker that encode data by identifying a set of colors at a corresponding set of predetermined locations on the visual marker. Then, the data of the visual marker is decoded using the colored markings and the set of colors of the color-interpretation scheme.

Growing community of inventors

Oakland, CA, United States of America

Tom Sengelaub