Meta Platforms Technologies, LLC

Nathan Aschenbach

Pol Pla I Conesa

Aaron S Faucher

Daniel Rosas

Chris Rojas

Bryan Sparks

Emron Jackson Henry

Johnathon Simmons

Justin Strawn

Aspects of the present disclosure are directed to triggering virtual keyboard selections using multiple input modalities. An interface manager can display an interface, such as a virtual keyboard, to a user in an artificial reality environment. Implementations of the interface manager can track user eye gaze input and user hand input (e.g., hand or finger motion). The interface manager can resolve a character selection on the virtual keyboard according to the tracked user gaze input based on detection that the user's hand motion meets a trigger criteria. For example, the interface manager can: detect that the tracked user hand motion meets the trigger criteria at a given point in time; and resolve a selection from the virtual keyboard (e.g., selection of a displayed character) according to the tracked user gaze on the virtual keyboard at the given point in time.

Virtual selections using multiple input modalities

Mixed reality (MR) user interactions are enabled through a combination of eye tracking (user gaze determination) and secondary inputs such as finger gestures, hand gestures, eye gestures, wrist band device input, handheld controller input, and similar ones. A location of interest in displayed content may be identified through the user's gaze, fixation, and/or saccades, or other actions, such as zoom, rotate, pan, move, open actionable menus, etc., may be performed on the location of interest based on the secondary inputs.

Mixed reality interaction with eye-tracking techniques

Aspects of the disclosure are directed to an interface for receiving input using multiple modalities in an artificial reality environment. The interface can be a virtual keyboard displayed in an artificial reality environment that includes characters arranged as elements. Implementations include an artificial reality device/system for displaying the artificial reality environment and receiving user input in a first modality, and a controller device for receiving user input in an additional input modality. For example, the artificial reality system can be configured to receive user gaze input as a first input modality and the controller device can be configured to receive input in a second modality, such as touch input received at a trackpad. An interface manager can process input in the multiple modalities to control an indicator on the virtual interface. The interface manager can also resolve character selections from the virtual interface according to the input.

Artificial reality input using multiple modalities

Virtual keyboard selections using multiple input modalities

Growing community of inventors

Seattle, WA, United States of America

Nathan Aschenbach