Apple Inc.

Ali Ercan

Julian K Shutzberg

David J Meyer

Paul A Lacey

Emmanuel Piuze-Phaneuf

Chase B Lortie

David M Teitelbaum

Wei-Kang Hsu

Various implementations assess user interactions to trigger user interface responses. Some implementations associate user gaze with user interface elements using one or more criteria that enable interpreting the user's gaze in ways that are consistent with user intentions and/or expectations. In some implementations, a user's gaze is associated with a user interface element when the user gazes at the user interface element. When the gaze leaves the user interface element (e.g., intentionally or unintentionally), the association may be maintained or discontinued using one or more criteria to be consistent with user intentions and/or expectations. In some implementations, this involves determining whether to continue associating a user's gaze with a user interface element so that the gaze-to-element association is maintained in certain circumstances (e.g., blinks and saccades) but discontinued in other circumstances (e.g., when the user is fixating outside of a zone around the element for more than a threshold time/number of frames).

Optimizations in associating user gaze with user interface elements

Various implementations identify user input based on the intended display frame rate and the adjusted user data. For example, an example process may include obtaining user data associated with an activity of a user via a sensor in a physical environment. The process may further include obtaining an intended display frame rate corresponding to one or more portions of a user interface being presented within an extended reality (XR) environment, where the intended display frame rate includes an irregular timing between frames. The process may further include adjusting the user data based on the irregular timing of the intended display frame rate. The process may further include identifying user input to the user interface based on the intended display frame rate and the adjusted user data.

Adjusting user data based on a display frame rate

In some implementations, a method includes: determining a location for virtual content; detecting a user interaction with the virtual content; in response to detecting the user interaction with the virtual content, determining a position of a hand gesture during the user interaction with the virtual content; in accordance with a determination that the position of the hand gesture is within a threshold distance relative to the location of the virtual content, generating corrected hand tracking data associated with the user interaction with the virtual content; and in accordance with a determination that the position of the hand gesture is outside of the threshold distance relative to the location of the virtual content, generating uncorrected hand tracking data associated with the user interaction with the virtual content.

Heuristic for hand tracking correction modes

In some implementations, a method includes: obtaining uncorrected hand tracking data; obtaining a depth map associated with a physical environment; identifying a position of a portion of the finger within the physical environment based on the depth map and the uncorrected hand tracking data; performing spatial depth smoothing on a region of the depth map adjacent to the position of the portion of the finger; and generating corrected hand tracking data by performing point of view (POV) correction on the uncorrected hand tracking data based on the spatially depth smoothed region of the depth map adjacent to the portion of the finger.

Growing community of inventors

San Jose, CA, United States of America

Ali Ercan