Microsoft Technology Licensing, LLC

Sheng Kai Tang

Julia Schwarz

Nicholas Ferianc Kamuda

Casey Leon Meekhof

Sophie Stellmach

Nahil Tawfik Sharkasi

Joshua Kyle Neff

Thomas Matthew Gable

Michael Harley Notter

Ramiro Torres

Mathew Julian Lamb

Jamie Bryant Kirschenbaum

Anthony James Ambrus

Austin Seungmin Lee

Alton Kwok

Jason Michael Ray

Adam Edwin Behringer

Kenneth Mitchell Jakubzak

Jenny Kam

Amy Mun Hong

Noe Moreno Barragan

Kevin John Appel

Amol Ashok Ambardekar

Pedro Urbina Escos

Ning Xu

Andrey Konin

Muhammad Zeeshan Zia

William Douglas Guyman

Harpreet Singh Sawhney

Keiichi Matsuda

Jonathan Palmer

Chuan Qin

Neil Richard Kronlage

Alon Farchy

Aleksandar Josic

Bilha-Catherine W Githinji

Ramiro S Torres

A computer implemented method includes detecting user interaction with mixed reality displayed content in a mixed reality system. User focus is determined as a function of the user interaction based on the user interaction using a spatial intent model. A length of time for extending voice engagement with the mixed reality system is modified based on the determined user focus. Detecting user interaction with the displayed content may include tracking eye movements to determine objects in the displayed content at which the user is looking and determining a context of a user dialog during the voice engagement.

Spatial attention model enhanced voice engagement system

Examples are disclosed that relate to hand gesture-based emojis. One example provides, on a display device, a method comprising receiving hand tracking data representing a pose of a hand in a coordinate system, based on the hand tracking data, recognizing a hand gesture, and identifying an emoji corresponding to the hand gesture. The method further comprises presenting the emoji on the display device, and sending an instruction to one or more other display devices to present the emoji.

Hand gesture-based emojis

A computing system is provided. The computing system includes a head mounted display (HMD) device including a display, a processor configured to execute one or more programs, and associated memory. The processor is configured to display a virtual object at least partially within a field of view of a user on the display, identify a plurality of control points associated with the virtual object, and determine that one or more of the control points associated with the virtual object are further than a predetermined threshold distance from the user. The processor is configured to, based on the determination, invoke a far interaction mode for the virtual object and receive a trigger input from the user. In response to the trigger input in the far interaction mode, the processor is configured to invoke a near interaction mode and display a virtual interaction object within the predetermined threshold distance from the user.

Near interaction mode for far virtual object

A method for single-handed microgesture input comprises receiving hand tracking data for a hand of a user. A set of microgesture targets that include software functions are assigned to positions along a length of a first finger. The received hand tracking data is analyzed by a gesture recognition machine. A location of a thumbtip of the hand of the user is determined relative to the positions along the first finger. Responsive to determining that the thumbtip is within a threshold distance of the first finger at a first position along the length of the first finger, a corresponding first microgesture target is designated for selection. Selection of the first microgesture target is enabled based on a duration the thumbtip is at the first position. Responsive to detecting a confirmation action, the corresponding microgesture target executes.

Single-handed microgesture inputs

This disclosure relates to displaying a user interface for a computing device based upon a user intent determined via a spatial intent model. One example provides a computing device comprising a see-through display, a logic subsystem, and a storage subsystem. The storage subsystem comprises instructions executable by the logic machine to receive, via an eye-tracking sensor, eye tracking samples each corresponding to a gaze direction of a user, based at least on the eye tracking samples, determine a time-dependent attention value for a location in a field of view of the see-through display, based at least on the time-dependent attention value for the location, determine an intent of the user to interact with a user interface associated with the location that is at least partially hidden from a current view, and in response to determining the intent, display via the see-through display the user interface.

Determining user intent based on attention values

A head-mounted display comprises a display device and an outward-facing depth camera. A storage machine comprises instructions executable by a logic machine to present one or more virtual objects on the display device, to receive information from the depth camera about an environment, and to determine a position of the head-mounted display within the environment. Based on the position of the head-mounted display, a position of a joint of a user's arm is inferred. Based on the information received from the depth camera, a position of a user's hand is determined. A ray is cast from a portion of the user's hand based on the position of the joint of the user's arm and the position of the user's hand. Responsive to the ray intersecting with one or more control points of a virtual object, the user is provided with an indication that the virtual object is being targeted.

Holographic palm raycasting for targeting virtual objects

Systems and methods are provided for selectively enabling or disabling control rays in mixed-reality environments. In some instances, a mixed-reality display device presents a mixed-reality environment to a user which includes one or more holograms. The display device then detects a user gesture input associated with a user control (which may include a part of the user's body) during presentation of the mixed-reality environment. In response to detecting the user gesture, the display device selectively generates and displays a corresponding control ray as a hologram rendered by the display device extending away from the user control within the mixed-reality environment. Gestures may also be detected for selectively disabling control rays so that they are no longer rendered.

Discrete and continuous gestures for enabling hand rays

A method for augmenting a two-stage hand gesture input comprises receiving hand tracking data for a hand of a user. A gesture recognition machine recognizes that the user has performed a first-stage gesture based on one or more parameters derived from the received hand tracking data satisfying first-stage gesture criteria. An affordance cueing a second-stage gesture is provided to the user responsive to recognizing the first-stage gesture. The gesture recognition machine recognizes that the user has performed the second-stage gesture based on one or more parameters derived from the received hand tracking data satisfying second-stage gesture criteria. A graphical user interface element is displayed responsive to recognizing the second-stage gesture.

Augmented two-stage hand gesture input

A method for single-handed microgesture input comprises receiving hand tracking data for a hand of a user. A set of microgesture targets that include software functions are assigned to positions along a length of a first finger. A visual affordance including indicators for two or more assigned microgesture targets is provided to the user. The received hand tracking data is analyzed by a gesture recognition machine. A location of a thumbtip of the hand of the user is determined relative to the positions along the first finger. Responsive to determining that the thumbtip is within a threshold distance of the first finger at a first position along the length of the first finger, an indicator for a corresponding first microgesture target is augmented, and then further augmented based on a duration the thumbtip is at the first position. Responsive to detecting a confirmation action, the corresponding microgesture target executes.

A method for object recognition includes, at a computing device, receiving an image of a real-world object. An identity of the real-world object is recognized using an object recognition model trained on a plurality of computer-generated training images. A digital augmentation model corresponding to the real-world object is retrieved, the digital augmentation model including a set of augmentation-specific instructions. A pose of the digital augmentation model is aligned with a pose of the real-world object. An augmentation is provided, the augmentation associated with the real-world object and specified by the augmentation-specific instructions.

Real-world object recognition for computing device

Systems and methods are provided for controlling the position of an interactive movable menu in a mixed-reality environment. In some instances, a mixed-reality display device presents a mixed-reality environment to a user. The mixed-reality device then detects a first gesture associated with a user controller while presenting the mixed-reality environment and, in response to the first gesture, triggers a display of an interactive movable menu within the mixed-reality environment as a tethered hologram that is dynamically moved within the mixed-reality environment relative to and corresponding with movement of the user controller within the mixed-reality environment. Then, in response to a second detected gesture, the mixed-reality device selectively locks a display of the interactive movable menu at a fixed position that is not tethered to the user controller.

Context-aware system menu behavior for mixed reality

A method for improving user interaction with a virtual environment includes presenting the virtual environment to a user on a display, measuring a gaze location of a user's gaze relative to the virtual environment, casting an input ray from an input device, measuring an input ray location at a distal point of the input ray, and snapping a presented ray location to the gaze location when the input ray location is within a snap threshold distance of the input ray location.

Systems and methods for using eye gaze to bend and snap targeting rays for remote interaction

Growing community of inventors

Redmond, WA, United States of America

Sheng Kai Tang