Microsoft Technology Licensing, LLC

Thomas Matthew Gable

Julia Schwarz

Casey Leon Meekhof

Sheng Kai Tang

Nahil Tawfik Sharkasi

Nicholas Ferianc Kamuda

Sophie Stellmach

Ramiro Torres

Jamie Bryant Kirschenbaum

Joshua Kyle Neff

Ivan Jelev Tashev

Alton Kwok

Jason Michael Ray

Yu-Te Wang

Kevin John Appel

Tan Gemicioglu

Raymond Michael Winters, Iv

Adam Edwin Behringer

Erian Vazquez

Kristian Jose Davila

Chuan Qin

Neil Richard Kronlage

Ramiro S Torres

This document relates to employing tongue gestures to control a computing device, and training machine learning models to detect tongue gestures. One example relates to a method or technique that can include receiving one or more motion signals from an inertial sensor. The method or technique can also include detecting a tongue gesture based at least on the one or more motion signals, and outputting the tongue gesture.

Inertial sensing of tongue gestures

A computing system is provided. The computing system includes a head mounted display (HMD) device including a display, a processor configured to execute one or more programs, and associated memory. The processor is configured to display a virtual object at least partially within a field of view of a user on the display, identify a plurality of control points associated with the virtual object, and determine that one or more of the control points associated with the virtual object are further than a predetermined threshold distance from the user. The processor is configured to, based on the determination, invoke a far interaction mode for the virtual object and receive a trigger input from the user. In response to the trigger input in the far interaction mode, the processor is configured to invoke a near interaction mode and display a virtual interaction object within the predetermined threshold distance from the user.

Near interaction mode for far virtual object

The techniques disclosed herein provide systems that can control the invocation of precision input mode. A system can initially utilize a first input device, such as a head-mounted display device monitoring the eye gaze direction of a user to control the location of an input target. When one or more predetermined input gestures are detected, the system can then invoke a precision mode that transitions the control of the input target from the first input device to a second input device. The second device can include another input device utilizing different input modalities, such as a sensor detecting one or more hand gestures of the user. The predetermined input gestures can include a fixation input gesture, voice commands, or other gestures that may include the use of a user's hands or head. By controlling the invocation of precision input mode using specific gestures, a system can mitigate device coordination issues.

Controlled invocation of a precision input mode

A head-mounted display comprises a display device and an outward-facing depth camera. A storage machine comprises instructions executable by a logic machine to present one or more virtual objects on the display device, to receive information from the depth camera about an environment, and to determine a position of the head-mounted display within the environment. Based on the position of the head-mounted display, a position of a joint of a user's arm is inferred. Based on the information received from the depth camera, a position of a user's hand is determined. A ray is cast from a portion of the user's hand based on the position of the joint of the user's arm and the position of the user's hand. Responsive to the ray intersecting with one or more control points of a virtual object, the user is provided with an indication that the virtual object is being targeted.

Holographic palm raycasting for targeting virtual objects

A method for augmenting a two-stage hand gesture input comprises receiving hand tracking data for a hand of a user. A gesture recognition machine recognizes that the user has performed a first-stage gesture based on one or more parameters derived from the received hand tracking data satisfying first-stage gesture criteria. An affordance cueing a second-stage gesture is provided to the user responsive to recognizing the first-stage gesture. The gesture recognition machine recognizes that the user has performed the second-stage gesture based on one or more parameters derived from the received hand tracking data satisfying second-stage gesture criteria. A graphical user interface element is displayed responsive to recognizing the second-stage gesture.

Augmented two-stage hand gesture input

A method for improving user interaction with a virtual environment includes presenting the virtual environment to a user on a display, measuring a gaze location of a user's gaze relative to the virtual environment, casting an input ray from an input device, measuring an input ray location at a distal point of the input ray, and snapping a presented ray location to the gaze location when the input ray location is within a snap threshold distance of the input ray location.

Systems and methods for using eye gaze to bend and snap targeting rays for remote interaction

Growing community of inventors

Seattle, WA, United States of America

Thomas Matthew Gable