Microsoft Technology Licensing, LLC

Casey Leon Meekhof

Julia Schwarz

Sheng Kai Tang

Sophie Stellmach

Nahil Tawfik Sharkasi

Jinyu Li

Tommer Leyvand

Nicholas Ferianc Kamuda

Thomas Matthew Gable

Jian Qiao Sun

Yichen Wei

Ramiro Torres

Craig C Peeper

Robert Matthew Craig

Ketan K Dalal

Vladimir Tankovich

Mitchell Stephen Dernis

Joshua Kyle Neff

Jason Michael Ray

Jamie Bryant Kirschenbaum

Tim Keosababian

Timothy Milton Keosababian

Eddie L Mays

Simon Stachniak

Alton Kwok

Richard Henry Irving

Bhaven P Dedhia

Russ Glaser

Bugra Tekin

Erian Vazquez

Ryan Gunn

Toan Huynh

Kevin John Appel

James Waletzky

Fabian Gobel

Ajay Ramjee

J Andrew Goossen

Fang Wen

Zsolt Mathe

Jason Matthew Gould

Michael E Pietraszak

David Bruce Lindsay

Alan Stuart

Patrick Orville Cook

Chuan Qin

Neil Richard Kronlage

Alon Farchy

Anton Rakovchuk

Kyle Nicholas San

Ramiro S Torres

A method for evaluating gesture input comprises receiving input data for sequential data frames, including hand tracking data for hands of a user. A first neural network is trained to recognize features indicative of subsequent gesture interactions and configured to evaluate input data for a sequence of data frames and to output an indication of a likelihood of the user performing gesture interactions during a predetermined window of data frames. A second neural network is trained to recognize features indicative of whether the user is currently performing one or more gesture interactions and configured to adjust parameters for gesture interaction recognition during the predetermined window based on the indicated likelihood. The second neural network evaluates the predetermined window for performed gesture interactions based on the adjusted parameters, and outputs a signal as to whether the user is performing one or more gesture interactions during the predetermined window.

Gesture recognition based on likelihood of interaction

A computing system is provided. The computing system includes a head mounted display (HMD) device including a display, a processor configured to execute one or more programs, and associated memory. The processor is configured to display a virtual object at least partially within a field of view of a user on the display, identify a plurality of control points associated with the virtual object, and determine that one or more of the control points associated with the virtual object are further than a predetermined threshold distance from the user. The processor is configured to, based on the determination, invoke a far interaction mode for the virtual object and receive a trigger input from the user. In response to the trigger input in the far interaction mode, the processor is configured to invoke a near interaction mode and display a virtual interaction object within the predetermined threshold distance from the user.

Near interaction mode for far virtual object

A head-mounted display comprises a display device and an outward-facing depth camera. A storage machine comprises instructions executable by a logic machine to present one or more virtual objects on the display device, to receive information from the depth camera about an environment, and to determine a position of the head-mounted display within the environment. Based on the position of the head-mounted display, a position of a joint of a user's arm is inferred. Based on the information received from the depth camera, a position of a user's hand is determined. A ray is cast from a portion of the user's hand based on the position of the joint of the user's arm and the position of the user's hand. Responsive to the ray intersecting with one or more control points of a virtual object, the user is provided with an indication that the virtual object is being targeted.

Holographic palm raycasting for targeting virtual objects

Examples that relate to virtual controls in a mixed reality experience are described. One example provides a method comprising, via a mixed reality display device, displaying mixed reality content including a representation of a virtual control, and receiving sensor data indicating motion of a user digit. The method further comprises, based at least in part on the sensor data, determining a velocity of the user digit, and responsive to determining that the velocity of the user digit relative to a surface corresponding to the virtual control satisfies a velocity-based selection condition, triggering the virtual control.

Velocity-based controls

Systems and methods are provided for selectively enabling or disabling control rays in mixed-reality environments. In some instances, a mixed-reality display device presents a mixed-reality environment to a user which includes one or more holograms. The display device then detects a user gesture input associated with a user control (which may include a part of the user's body) during presentation of the mixed-reality environment. In response to detecting the user gesture, the display device selectively generates and displays a corresponding control ray as a hologram rendered by the display device extending away from the user control within the mixed-reality environment. Gestures may also be detected for selectively disabling control rays so that they are no longer rendered.

Discrete and continuous gestures for enabling hand rays

Systems and methods are provided for detecting user-object interaction in mixed-reality environments. A mixed-reality system detects a controller gesture with an associated controller orientation in the mixed-reality environment. The mixed-reality system then determines an interaction region for the controller gesture and identifies one or more virtual objects within the interaction region. The virtual objects each have an associated orientation affinity. Subsequently, the mixed-reality system determines an orientation similarity score between the controller orientation and the orientation affinity for each virtual object within the interaction region. In response to determining that at least one orientation similarity score exceeds a predetermined threshold, the mixed-reality system executes an interaction between the controller and the virtual object that has the greatest orientation similarity score.

Hand motion and orientation-aware buttons and grabbable objects in mixed reality

A method for augmenting a two-stage hand gesture input comprises receiving hand tracking data for a hand of a user. A gesture recognition machine recognizes that the user has performed a first-stage gesture based on one or more parameters derived from the received hand tracking data satisfying first-stage gesture criteria. An affordance cueing a second-stage gesture is provided to the user responsive to recognizing the first-stage gesture. The gesture recognition machine recognizes that the user has performed the second-stage gesture based on one or more parameters derived from the received hand tracking data satisfying second-stage gesture criteria. A graphical user interface element is displayed responsive to recognizing the second-stage gesture.

Augmented two-stage hand gesture input

Systems and methods are provided for controlling the position of an interactive movable menu in a mixed-reality environment. In some instances, a mixed-reality display device presents a mixed-reality environment to a user. The mixed-reality device then detects a first gesture associated with a user controller while presenting the mixed-reality environment and, in response to the first gesture, triggers a display of an interactive movable menu within the mixed-reality environment as a tethered hologram that is dynamically moved within the mixed-reality environment relative to and corresponding with movement of the user controller within the mixed-reality environment. Then, in response to a second detected gesture, the mixed-reality device selectively locks a display of the interactive movable menu at a fixed position that is not tethered to the user controller.

Context-aware system menu behavior for mixed reality

A method for improving user interaction with a virtual environment includes presenting the virtual environment to a user on a display, measuring a gaze location of a user's gaze relative to the virtual environment, casting an input ray from an input device, measuring an input ray location at a distal point of the input ray, and snapping a presented ray location to the gaze location when the input ray location is within a snap threshold distance of the input ray location.

Systems and methods for using eye gaze to bend and snap targeting rays for remote interaction

A method for improving user interaction with a virtual environment includes measuring a first position of the user's gaze relative to a virtual element, selecting the virtual element in the virtual environment at an origin when the user's gaze overlaps the virtual element, measuring a second position of a user's gaze relative to the virtual element, presenting a visual placeholder at a second position of the user's gaze when the second position of the user's gaze is beyond a threshold distance from the origin, and moving the visual placeholder relative to a destination using a secondary input device.

Systems and methods for gaze-informed target manipulation

A candidate human head is found in depth video using a head detector. A head region of light intensity video is spatially resolved with a three-dimensional location of the candidate human head in the depth video. Facial recognition is performed on the head region of the light intensity video using a face recognizer.

Identification using depth-based head-detection data

Growing community of inventors

Redmond, WA, United States of America

Casey Leon Meekhof