Apple Inc.

Tom Sengelaub

Hua Gao

Hao Qin

Anselm Grundhoefer

Mohamed Selim Ben Himane

Thomas G Salter

Arun Srivatsan Rangaprasad

Jeffrey S Norris

Paul Ewers

Scott G Wade

Chia-Yin Tsai

Jie Zhong

Jonathan P Ive

Alan C Dye

Stephen Olivier Lemay

Julian Jaede

Fletcher R Rothkopf

Cheng Kai Chen

Jeremy C Franklin

William A Sorrentino, Iii

Jason C Sauers

Geoffrey Grant Stahl

Arthur Yasheng Zhang

Wey-Jiun Lin

Phil M Hobson

Ivan S Maric

Graham Butler Myhre

Michael J Rockwell

Forrest C Wang

Yoonhoo Jo

Muhammad F Hossain

Ritu Shah

Jia Tao

Marinus Meursing

Yanghai Tsin

Bertrand Nepveu

Kathrin Berkner Cieslicki

Ryan J Dunn

Julian Hoenig

Jan K Quijalvo

Jason L Slupeiks

Branko Petljanski

Lionel E Edwin

Peter Kaufmann

Martin Haller

Harihar Narasimha-Iyer

Thibaut Weise

James W Vandyke

Tobias Eble

Hesam Najafi

Marvin Klinkhardt

Uma Arunachalam

Rene Heideklang

Methods for performing a camera calibration process for outward-facing cameras on devices such as head-mounted display devices are disclosed. Using a second device, a calibration process for the outward-facing cameras on a first device may be performed. Communications of information for a calibration scheme comprising a ground truth comprising features and respective distance separating the features currently displayed or to be displayed may be performed between the first and second device. Calibration parameters may be determined by comparing identified features in captured images of the calibration scheme displayed on the second device to the ground truth.

Infield camera calibration using display device

Methods to trigger eye enrollment without requiring good gaze interaction are described that allow a guest user of a device to initiate partial or full eye enrollment even though their eye model is not known and thus conventional gaze-based interactions do not work well. A gaze tracking system collects gaze data in the background. At any time (or within an interval after a user puts on the device), an eye enrollment can be triggered by detecting some gaze gesture, for example rolling the eyes in a large circle, or moving the eyes randomly for a time that exceeds a threshold. Depending on the coverage of the gaze/cornea data collected in the background, a full eye enrollment or only a visual axis enrollment may be performed in response to the gesture.

Gesture-initiated eye enrollment

Methods and apparatus for generating user-aware eye models. During an enrollment process, images of a user's eye are captured by one or more cameras when the eye is in two or more different orientations and at two or more different levels of display brightness. The captured images are processed to generate a 3-dimensional, user-aware eye model, for example a model of at least the eye's cornea and pupil features. The generated user-aware eye model may be used in other processes, for example in a gaze tracking process. The enrollment process may be an iterative process to optimize the eye model, or a continuous process performed while the user is using the system.

Eye model enrollment

Various implementations disclosed herein include an inconspicuous eye enrollment for user devices through the simultaneous or near-simultaneous eye characteristic tracking and eye model updating. The implementations disclosed herein direct light towards an eye of a user to produce glint reflections and obtain, via one or more sensors, image frames comprising depictions of the glint reflections. Based on those image frames, the implementations disclosed herein track eye characteristics and update an eye model of the eye during a period of time between obtaining a first image frame of the image frames and obtaining a second image frame of the image frames.

Methods, systems, and apparatuses for simultaneous eye characteristic tracking and eye model updating

Low-power eye tracking for detecting position and movements of a user's eyes in a head-mounted device (HMD). An eye tracking system for an HMD may include eye tracking cameras and eye odometers that may reduce power consumption of the system. The eye odometers may be used as a low-power component to track relative movement of the user's eyes with respect to the HMD between frames captured by the eye tracking cameras, which may allow the frame rate of the eye tracking cameras to be reduced.

Eye tracking using eye odometers

Methods and apparatus for flexible illumination that improve the performance and robustness of an imaging system are described. Multiple different lighting configurations for the imaging system are pre-generated. Each lighting configuration may specify one or more aspects of lighting. A lookup table may be generated via which each pose is associated with a respective lighting configuration. A user may put on, hold, or otherwise use the device. A process may be initiated in which different lighting configurations may be selected by the controller to capture images of the user's eye, periorbital region, or face at different poses and in different conditions for use by a biometric authentication or gaze tracking process.

Flexible illumination for imaging systems

In an unobtrusive visual axis enrollment process, a line of text or other content is displayed at a known vertical location and virtual depth that the user can then read. This line of text may be content that the user needs to read as part of the normal enrollment process. As the user reads the line of text, eye tracking cameras may capture images of the eye. This data may then be used to estimate a stimulus plane. The error between the estimated stimulus plane and the ground truth stimulus plane (the actual location of the line of text in virtual space) may then be used to estimate the kappa angle.

Visual axis enrollment

A personalized eye model is used to generate synthetic gaze features at ground-truth eye poses G. Corresponding synthetic gaze poses Gare estimated from the synthetic gaze features using an average eye model. A linear regression is applied between Gand Gto generate a gaze correction function. The gaze correction function represents differences between the synthetic gaze Gof the subject eye at the display and that of the average eye model Gat the display, but does not contain security- or privacy-sensitive information. Further, the personalized eye model cannot be recovered from the gaze correction function, and thus the gaze correction function can be stored unencrypted and available for use during a cold boot of a device prior to login. On a cold boot of the device, the gaze correction function may be accessed and used with an average eye model to improve gaze-based interactions.

Synthetic gaze enrollment

Systems and methods are disclosed to enable detection of excessive contact lens shift in a head-mounted display (HMD) device that implement gaze tracking for the user. In embodiments, during a user enrollment process to calibrate the gaze tracking system, the HMD device analyzes key frames of the eye to estimate a first center location of the eye based on detected eye features such as the pupil, the iris, or the limbus, and a second center location of the eye based on the shape of the detected cornea. Shift vectors are generated based on the two center locations for individual key frames. If the standard deviation of the vectors' magnitudes exceeds a specified threshold, excessive lens shift is detected. In response to detection of excessive contact lens shift, the HMD device may generate a notification to the user or activate compensation mechanisms.

Contact lens shift detection for head-mounted display devices

Head-mounted devices (HMDs) include display screens for display of video and/or images. A complex lens is positioned between the display screen and a user's eye to render images shown on the display screen for correct appearance to the user's eye. Changes in relative position or orientation between the display and the lens may cause unwanted changes in the appearance of the images viewed by the user. A process is disclosed to determine changes in relative positions and orientation of the display screen and lens. By determining changes in relative positions and orientation, recalibration of the system may be implemented to continually present images to the user's eye with the proper focus for a more satisfactory user experience.

Camera and display calibration

A head-mountable display device includes a housing defining a front opening and a rear opening, a display screen disposed in the front opening, a display assembly disposed in the rear opening, a first securement strap coupled to the housing, the first securement strap including a first electronic component, a second securement strap coupled to the housing, the second securement strap including a second electronic component, and a securement band extending between and coupled to the first securement strap and the second securement strap.

Head mountable display

Various implementations disclosed herein include devices, systems, and methods that provide detecting and decoding of visual markers that encode data. In some implementations, a visual marker is detected in an image of a physical environment. In some implementations, detecting the visual marker is based on a pre-determined portion of the visual marker. In some implementations, detecting the visual marker is based on a predefined shape. In some implementations, a visual marker interpretation scheme is determined for decoding the visual marker. In some implementations, the visual marker is decoded based on one or more colors being attributed a same value. In some implementations, the visual marker is decoded based on one or more other colors being attributed a different value.

Visual marker with user selectable appearance

Operating an outward-facing display device for low light conditions may include receiving information indicative of ambient lighting conditions for the display device's environment; in accordance with a determination that the ambient lighting conditions do not satisfy a brightness criterion, determining a set of pixels in the outward-facing display device corresponding to an area of interest in the environment; and driving the set of pixels at a particular brightness to improve lighting in the area of interest.

Electronic device with a display for low light conditions

Various implementations disclosed herein include devices, systems, and methods that assess calibration between eye tracking and other components of a head-mounted device (HMD) using another device, such as a mobile device. For example, an example process may include obtaining first sensor data captured by a first sensor of a first device, the first sensor data including a representation of a portion of a second device, obtaining second sensor data captured by a second sensor of the second device, detecting a position of the first sensor of the first device based on the second sensor data, and assessing a calibration between the portion of the second device and the sensor of the second device based on the first sensor data and the detected position of the first sensor.

Display calibration

Various implementations disclosed herein include devices, systems, and methods that provide color visual markers that include colored markings that encode data, where the colors of the colored markings are determined by scanning (e.g., detecting the visual marker using a sensor of an electronic device) the visual marker itself. In some implementations, a visual marker is detected in an image of a physical environment. In some implementations, the visual marker is detected in the image by detecting a predefined shape of a first portion of the visual marker in the image. Then, a color-interpretation scheme is determined for interpreting colored markings of the visual marker that encode data by identifying a set of colors at a corresponding set of predetermined locations on the visual marker. Then, the data of the visual marker is decoded using the colored markings and the set of colors of the color-interpretation scheme.

Growing community of inventors

Oakland, CA, United States of America

Tom Sengelaub