Apple Inc.

United States of America as Represented by the Administrator of Nasa

The United States of America as Represented by the Administrator of the

Other

Andrew B Watson

Mehmet N Agaoglu

Jiaying Wu

Yashas Rai Kurlethimar

Nicolas Pierre Marie Frederic Bonnier

Can Jin

Alexander G Berardino

Albert J Ahumada, Jr

Tim H W Cornelissen

Chaohao Wang

Yang Li

Nicolas P Bonnier

Seyedkoosha Mirhosseini

Alexandre Chapiro

Yi-Pai Huang

Elijah H Kleeman

Thomas Post

Tim H Cornelissen

Pretesh A Mascarenhas

Laura Lynn Walker

Laura L Walker

Various implementations disclosed herein include devices, systems, and methods that determine a gaze behavior state to identify gaze shifting events, gaze holding events, and loss events of a user based on physiological data. For example, an example process may include obtaining eye data associated with a gaze during a first period of time (e.g., eye position and velocity, interpupillary distance, pupil diameters, etc.). The process may further include obtaining head data associated with the gaze during the first period of time (e.g., head position and velocity). The process may further include determining a first gaze behavior state during the first period of time to identify gaze shifting events, gaze holding events, and loss events (e.g., one or more gaze and head pose characteristics may be determined, aggregated, and used to classify the user's eye movement state using machine learning techniques).

Gaze behavior detection

Approaches for presenting user cues based on distance between an electronic device and an object are described. In some examples, content is displayed. First sensor data is received, which is indicative of a first distance between the electronic device and the object. A first version of a user cue relating to the first distance may be presented. Second sensor data is received, which is indicative of a second distance between the electronic device and the object. A second version of the user cue relating to the second distance may be presented.

Near-viewing notification techniques

A vertical lookup table (LUT) stores first vertical scale values for a first display in a first portion and second vertical scale values for a second display in a second portion. A horizontal LUT stores horizontal scale values for the first display and the second display. A first version of visual content is generated for the first display by applying the first vertical scale values and the horizontal scale values to a first image frame that represents the visual content at a first resolution. A second version of the visual content is generated for the second display by applying the second vertical scale values and the horizontal scale values to a second image frame that represents the visual content at a second resolution.

Generating different versions of visual content for display

An electronic device may have a display and a gaze-tracking system. The display may use different resolutions in different portions of the display. Point-of-gaze information from the gaze-tracking system may be used to align a high-resolution portion of the display with the viewer's point-of-gaze. In the surrounding portions of the display, the resolution of the display may be decreased. This reduces power consumption, computation power, and bandwidth requirements without sacrificing the user experience. The resolution of the human eye is lower for purely chromatic signals than it is for luminance (brightness signals). To exploit this, the image data for the display may have an achromatic (white/black) luminance channel and two chromatic opponent channels. The opponent channels may then be compressed by a greater amount than the luminance channel, which increases the overall compression of the image data.

Display with chromatic foveation

Tracking an eye characteristic (e.g., gaze direction or pupil position) of a user's eyes by staggering image capture and using a predicted relationship between the user's eyes between eye captures to predict that eye's eye characteristic between those eye captures. Images of a user's eyes are captured in a staggered manner in the sense that the images of second eye are captured between the capture times of the images of the first eye and vice versa. An eye characteristic of the first eye at the capture times is determined based on the images of the first eye at those times. In addition, the eye characteristic of that first eye is predicted at additional times between captures based on a predicted relationship between the eyes.

Eye tracking using efficient image capture and vergence and inter-pupillary distance history

An electronic device may have a display for displaying image content. Head-mounted support structures in the device may be used to support the display. The electronic device may have an eye monitoring system that detects eye saccades and eye blinks. Control circuitry in the electronic device may coordinate operation of the display with periods of suppressed visual sensitivity that are associated with the saccades and blinks. By making adjustments to display circuitry and image content during periods of suppressed visual sensitivity, potentially visually obtrusive changes to displayed images can be hidden from a user of the electronic device. Adjustments to display operation may help reduce burn-in effects, may help reduce power consumption, and may otherwise improve device performance.

Electronic devices with display operation based on eye activity

An electronic device that includes a display and an eye tracker configured to collect eye tracking data regarding a gaze of one or more of a user's eyes across the display is disclosed herein. The electronic device includes processing circuitry that is operatively coupled to the display and configured to foveate one or more areas of the display according to the eye tracking data. If the eye tracking data input is lost, the processing circuitry is configured to recover from the loss of eye tracking data by changing one or more aspects of the foveated areas (e.g., size, resolution, etc.) until a threshold is satisfied. As time elapses since loss of eye tracking, the foveated areas move toward a center or a salient region of the display.

Recovery from eye-tracking loss in foveated displays

In an embodiment, an electronic device includes a display and an eye tracker. The display includes one or more foveated areas. In the embodiment, the eye tracker is configured to collect eye tracking data regarding a gaze of one or more eyes of a user on the display. The electronic device also includes processing circuitry operatively coupled to the display. In the embodiment, the processing circuitry is configured to receive an indication of a motion associated with the gaze from the eye tracker. The processing circuitry is also configured to determine a previous location associated with the gaze during a previous frame and a target position associated with the gaze during a target frame. In the embodiment, the processing circuitry is configured to expand one or more foveated areas of the display adjacent a previous position of the gaze of the user.

Peripheral luminance or color remapping for power saving

Techniques are provided for implementing near-view notification techniques. In some instances, a viewing distance with respect to a display screen of an electronic device may be determined. In accordance with a determination that the viewing distance is above a distance threshold, content may be presented on the display screen. In accordance with a determination that the viewing distance is under a distance threshold, until the viewing distance returns to being above the distance threshold, a visual notification may be displayed. The visual notification may disrupt the content presented on the display screen.

An electronic device may have a foveated display, an eye-tracking system and a head movement detection system. The eye-tracking system may gather information on a user's point of regard on the display while the head movement detection system may capture information regarding the rotation of the observer's head. Based on the point-of-regard information, head rotation information, image data, the type of eye/head movement that is underway, and/or tiredness information, control circuitry in the electronic device may produce image data for a display, with areas of different resolutions and(or) visual quality. A full-resolution and(or) quality portion of the image may overlap the point of regard. One or more lower resolution portions of the image may surround the full-resolution portion. The control circuitry may include a gaze prediction system for predicting the movement of the user's gaze during a saccade.

Electronic device with foveated display and gaze prediction

A method of measuring the video acuity of a physical imaging system is disclosed. A chart comprising an image of a set of test symbols is provided. The set of test symbols comprises a plurality of symbols repeated at a plurality of symbol sizes. A digital image of the chart is obtained using the imaging system. The digital image is cropped to obtain a set of cropped images comprising individual digital images of each test symbol in the set of test symbols. A template image of a test symbol is aligned with a test image comprising one member of the set of cropped images. The template image comprises a digitally generated image of the test symbol in the test image. The normalized correlation between the aligned template image and the test image is determined. The test symbol whose template image has the highest correlation with the test image is identified.

Growing community of inventors

Los Gatos, CA, United States of America

Andrew B Watson