Apple Inc.

David M Teitelbaum

Julian K Shutzberg

David J Meyer

Christopher D Mckenzie

Ali Ercan

Daniel J Brewer

Chase B Lortie

Lorenzo Soto Doblado

Bharat C Dandu

Various implementations disclosed herein include devices, systems, and methods that interpret user activity as user interactions with user interface (UI) elements positioned within a three-dimensional (3D) space such as an extended reality (XR) environment. Some implementations enable user interactions with virtual elements displayed in 3D environments that utilize alternative input modalities, e.g., XR environments that interpret user activity as either direct interactions or indirect interactions with virtual elements.

Input recognition based on distinguishing direct and indirect user interactions

Various implementations identify user input based on the intended display frame rate and the adjusted user data. For example, an example process may include obtaining user data associated with an activity of a user via a sensor in a physical environment. The process may further include obtaining an intended display frame rate corresponding to one or more portions of a user interface being presented within an extended reality (XR) environment, where the intended display frame rate includes an irregular timing between frames. The process may further include adjusting the user data based on the irregular timing of the intended display frame rate. The process may further include identifying user input to the user interface based on the intended display frame rate and the adjusted user data.

Adjusting user data based on a display frame rate

Various implementations disclosed herein include devices, systems, and methods that associate user activities with user interface (UI) elements within a 3D environment. Some implementations do so in ways that account for inaccuracies (i.e., fuzziness) in sensor-based detection of the user activities, e.g., inaccuracy in sensor data-based gaze tracking or sensor data-based hand/joint position. Some implementations use a sampling technique to associate user activity in a 3D environment with an appropriate portion of a UI positioned within a 3D environment. For example, a sampling technique may be used to identify sample locations within the 3D environment (e.g., sample locations around a gaze direction) to evaluate and ultimately select from to associate with a user activity, e.g., associating a particular UI button with a gaze direction.

Fuzzy hit testing

Growing community of inventors

San Francisco, CA, United States of America

David M Teitelbaum