Microsoft Technology Licensing, LLC

Mathew Julian Lamb

Alex Aben-Athar Kipman

Kathryn Stone Perez

Ben J Sugden

Robert L Crocco, Jr

Brian E Keane

Christopher E Miles

Laura K Massey

Anthony James Ambrus

Sophie Stellmach

Austin Seungmin Lee

Kenneth Mitchell Jakubzak

Amy Mun Hong

Stephen G Latta

Julia Schwarz

Sheng Kai Tang

Adam Edwin Behringer

Keiichi Matsuda

Jonathan Palmer

Daniel Joseph McCulloch

Arthur Charles Tomlin

Brian James Mount

Darren Bennett

Jason Bradley Scott

Ryan Lucas Hastings

Nicholas Ferianc Kamuda

Joshua Kyle Neff

Jenny Kam

Michael Harley Notter

Andrew David Wilson

Kevin Geisner

Jeffrey Neil Margolis

Alex A Kipman

Adam G Poulos

Benjamin Isaac Vaught

Jouya Jadidian

Jonathan Steed

Michael Scavezze

Dan B Kroymann

Tom G Salter

Sheridan Martin Small

Alton Kwok

Sebastian Sylvan

Jason Michael Ray

Lewey Geselowitz

Carlos Fernando Faria Costa

Erian Vazquez

Roger Sebastian-Kevin Sylvan

Kristian Jose Davila

Brian Thomas Merrell

Examples are disclosed that relate to utilizing a facial tracking sensor for controlling computer-generated facial expressions. One example provides a method of controlling computer-generated facial expressions. The method comprises receiving a sensor value acquired via a facial tracking sensor and determining an interpolated value for the sensor value within a value range. The value range corresponding to a blendshape range for a facial expression. The method further comprises determining a blendshape mapping based at least upon the interpolated value, determining expression data based at least upon the blendshape mapping, and providing the expression data to a device.

Controlling computer-generated facial expressions

A computer implemented method includes detecting user interaction with mixed reality displayed content in a mixed reality system. User focus is determined as a function of the user interaction based on the user interaction using a spatial intent model. A length of time for extending voice engagement with the mixed reality system is modified based on the determined user focus. Detecting user interaction with the displayed content may include tracking eye movements to determine objects in the displayed content at which the user is looking and determining a context of a user dialog during the voice engagement.

Spatial attention model enhanced voice engagement system

Systems and methods are provided for facilitating the presentation of expressive intent and other status information with messaging and other communication applications. The expressive intent is based on expressive effect data associated with the message recipients and/or message senders. The expressive intent can be conveyed through avatars and modified message content. The avatars convey gestures, emotions and other status information and the presentation of the avatars can be reactive to detected state information of the message recipient(s), message sender(s) and/or corresponding messaging device(s).

Modifying an avatar to reflect a user's expression in a messaging platform

Examples are disclosed that relate to hand gesture-based emojis. One example provides, on a display device, a method comprising receiving hand tracking data representing a pose of a hand in a coordinate system, based on the hand tracking data, recognizing a hand gesture, and identifying an emoji corresponding to the hand gesture. The method further comprises presenting the emoji on the display device, and sending an instruction to one or more other display devices to present the emoji.

Hand gesture-based emojis

Examples are disclosed that relate to utilizing image sensor inputs from different devices having different perspectives in physical space to construct an avatar of a first user in a video stream. The avatar comprises a three-dimensional representation of at least a portion of a face of the first user texture mapped onto a three-dimensional body simulation that follows actual physical movement of the first user. The three-dimensional body simulation of the first user is generated based on image data received from an imaging device and image sensor data received from a head-mounted display device both associated with the first user. The three-dimensional representation of the face of the first user is generated based on the image data received from the imaging device. The resulting video stream is sent, via a communication network, to a display device associated with a second user.

Multiple device sensor input based avatar

A computer-implemented method for determining focus of a user is provided. User input is received. An intention image of a scene including a plurality of interactive objects is generated. The intention image includes pixels encoded with intention values determined based on the user input. An intention value indicates a likelihood that the user intends to focus on the pixel. An intention score is determined for each interactive object based on the intention values of pixels that correspond to the interactive object. An interactive object of the plurality of interactive objects is determined to be a focused object that has the user's focus based on the intention scores of the plurality of interactive objects.

Intention image analysis for determining user focus

One example provides a computing device comprising instructions executable to receive information regarding one or more entities in the scene, to receive eye tracking a plurality of eye tracking samples, each eye tracking sample corresponding to a gaze direction of a user and, based at least on the eye tracking samples, determine a time-dependent attention value for each entity of the one or more entities at different locations in a use environment, the time-dependent attention value determined using a leaky integrator. The instructions are further executable to receive a user input indicating an intent to perform a location-dependent action, associate the user input to with a selected entity based at least upon the time-dependent attention value for each entity, and perform the location-dependent action based at least upon a location of the selected entity.

Location-based entity selection using gaze tracking

Examples are disclosed herein relating to displaying a virtual assistant. One example provides an augmented reality display device comprising a see-through display, a logic subsystem, and a storage subsystem storing instructions executable by the logic subsystem to display via the see-through display a virtual assistant associated with a location in a real-world environment, detect a change in a field of view of the see-through display, and when the virtual assistant is out of the field of view of the see-through display after the change in the field of view, display the virtual assistant in a virtual window on the see-through display.

Display of virtual assistant in augmented reality

Devices, systems, and methods for interacting with a three-dimensional virtual environment, including receiving an input associated with a change in pose of a user's hand; estimating, based on at least the input, a first pose in the virtual environment for an input source associated with the hand; identifying a surface of a virtual object in the virtual environment; rendering a frame depicting elements of the virtual environment, the frame including a pixel rendered for a position on the surface; determining a distance between the position and a virtual input line extending through a position of the first pose and in a direction of the first pose; changing a pixel color rendered for the pixel based on the distance between the position and the virtual input line; and displaying the frame including the pixel with the changed pixel color to the user via a head-mounted display device.

User interface discovery and interaction for three-dimensional virtual environments

A system for allowing a virtual object to interact with other virtual objects across different spaces within an augmented reality (AR) environment and to transition between the different spaces is described. An AR environment may include a plurality of spaces, each comprising a bounded area or volume within the AR environment. In one example, an AR environment may be associated with a three-dimensional world space and a two-dimensional object space corresponding with a page of a book within the AR environment. A virtual object within the AR environment may be assigned to the object space and transition from the two-dimensional object space to the three-dimensional world space upon the detection of a space transition event. In some cases, a dual representation of the virtual object may be used to detect interactions between the virtual object and other virtual objects in both the world space and the object space.

Mechanism to give holographic objects saliency in multiple spaces

A system and related methods for visually augmenting an appearance of a physical environment as seen by a user through a head-mounted display device are provided. In one embodiment, a virtual environment generating program receives eye-tracking information, lighting information, and depth information from the head-mounted display. The program generates a virtual environment that models the physical environment and is based on the lighting information and the distance of a real-world object from the head-mounted display. The program visually augments a virtual object representation in the virtual environment based on the eye-tracking information, and renders the virtual object representation on a transparent display of the head-mounted display device.

Virtual environment generating system

Technology is disclosed herein to help a user navigate through large amounts of content while wearing a see-through, near-eye, mixed reality display device such as a head mounted display (HMD). The user can use a physical object such as a book to navigate through content being presented in the HMD. In one embodiment, a book has markers on the pages that allow the system to organize the content. The book could have real content, but it could be blank other than the markers. As the user flips through the book, the system recognizes the markers and presents content associated with the respective marker in the HMD.

Navigating content using a physical object

Methods for enabling hands-free selection of objects within an augmented reality environment are described. In some embodiments, an object may be selected by an end user of a head-mounted display device (HMD) based on detecting a vestibulo-ocular reflex (VOR) with the end user's eyes while the end user is gazing at the object and performing a particular head movement for selecting the object. The object selected may comprise a real object or a virtual object. The end user may select the object by gazing at the object for a first time period and then performing a particular head movement in which the VOR is detected for one or both of the end user's eyes. In one embodiment, the particular head movement may involve the end user moving their head away from a direction of the object at a particular head speed while gazing at the object.

Radial selection by vestibulo-ocular reflex fixation

Growing community of inventors

Mercer Island, WA, United States of America

Mathew Julian Lamb