Microsoft Technology Licensing, LLC

Austin Seungmin Lee

Anthony James Ambrus

Mathew Julian Lamb

Brandon V Taylor

Mark Robert Swift

Keiichi Matsuda

Ryan S Menezes

Sophie Stellmach

Henry Yao-Tsu Chen

Kenneth Mitchell Jakubzak

Hauke Antony Gentzkow

Amy Mun Hong

Eddie Louis Mays, Iii

Lauren Diana Lo

Alice Shirong Ferng

Jason Thomas Faulkner

Jonathan Reed Harris

Sheng Kai Tang

Brian Scott Stucker

Jonathan Palmer

Shawn Crispin Wright

Alton Kwok

Hae Jin Lee

Albert Robles

Malek Mohamad Nafez Chalabi

Bofan Hsu

Aaron David Rogers

Becky Ann Brown

Siddhant Vipul Mehta

Rachel Fox Romano

Angela Chin

Cadin Lee Batrack

Sean Michael Lynch

Aleksandar Josic

Eddie Louis Iii Mays

A computer implemented method includes detecting user interaction with mixed reality displayed content in a mixed reality system. User focus is determined as a function of the user interaction based on the user interaction using a spatial intent model. A length of time for extending voice engagement with the mixed reality system is modified based on the determined user focus. Detecting user interaction with the displayed content may include tracking eye movements to determine objects in the displayed content at which the user is looking and determining a context of a user dialog during the voice engagement.

Spatial attention model enhanced voice engagement system

Systems and methods are provided for facilitating the presentation of expressive intent and other status information with messaging and other communication applications. The expressive intent is based on expressive effect data associated with the message recipients and/or message senders. The expressive intent can be conveyed through avatars and modified message content. The avatars convey gestures, emotions and other status information and the presentation of the avatars can be reactive to detected state information of the message recipient(s), message sender(s) and/or corresponding messaging device(s).

Modifying an avatar to reflect a user's expression in a messaging platform

This disclosure relates to displaying a user interface for a computing device based upon a user intent determined via a spatial intent model. One example provides a computing device comprising a see-through display, a logic subsystem, and a storage subsystem. The storage subsystem comprises instructions executable by the logic machine to receive, via an eye-tracking sensor, eye tracking samples each corresponding to a gaze direction of a user, based at least on the eye tracking samples, determine a time-dependent attention value for a location in a field of view of the see-through display, based at least on the time-dependent attention value for the location, determine an intent of the user to interact with a user interface associated with the location that is at least partially hidden from a current view, and in response to determining the intent, display via the see-through display the user interface.

Determining user intent based on attention values

Examples are disclosed that relate to utilizing image sensor inputs from different devices having different perspectives in physical space to construct an avatar of a first user in a video stream. The avatar comprises a three-dimensional representation of at least a portion of a face of the first user texture mapped onto a three-dimensional body simulation that follows actual physical movement of the first user. The three-dimensional body simulation of the first user is generated based on image data received from an imaging device and image sensor data received from a head-mounted display device both associated with the first user. The three-dimensional representation of the face of the first user is generated based on the image data received from the imaging device. The resulting video stream is sent, via a communication network, to a display device associated with a second user.

Multiple device sensor input based avatar

One example provides a computing device comprising instructions executable to receive information regarding one or more entities in the scene, to receive eye tracking a plurality of eye tracking samples, each eye tracking sample corresponding to a gaze direction of a user and, based at least on the eye tracking samples, determine a time-dependent attention value for each entity of the one or more entities at different locations in a use environment, the time-dependent attention value determined using a leaky integrator. The instructions are further executable to receive a user input indicating an intent to perform a location-dependent action, associate the user input to with a selected entity based at least upon the time-dependent attention value for each entity, and perform the location-dependent action based at least upon a location of the selected entity.

Location-based entity selection using gaze tracking

Systems, methods, and software are disclosed herein for enhancing the content capture experience on computing devices. In an implementation, a combined user input comprises a voice signal and a touch gesture sustained at least partially coincident with the voice signal. An occurrence of the combined user input triggers the identification of an associated content object which may then be associated with a captured version of the voice signal. Such an advance provides users with a new framework for interacting with their devices, applications, and surroundings.

Content capture experiences driven by multi-modal user inputs

Systems, methods, and software are disclosed herein for associating images with content items. In an implementation, a computing device determines that an image captured of a scene includes a display of meeting content. The device identifies a content item comprising a source of the meeting content in the scene and may then create an association between an instance of the image and an instance of the content item. The device may, for example, create a link to the content item that can be embedded in or otherwise associated with the image file. The reverse may also be accomplished, whereby a link to the image may be created and embedded in or otherwise associated with the content item.

Associating content items with images captured of meeting content

Examples are disclosed herein relating to displaying a virtual assistant. One example provides an augmented reality display device comprising a see-through display, a logic subsystem, and a storage subsystem storing instructions executable by the logic subsystem to display via the see-through display a virtual assistant associated with a location in a real-world environment, detect a change in a field of view of the see-through display, and when the virtual assistant is out of the field of view of the see-through display after the change in the field of view, display the virtual assistant in a virtual window on the see-through display.

Display of virtual assistant in augmented reality

Systems, methods, and software are disclosed herein that relate to user input combined from spatial gestures and voice. A combined user input allows users to interact more fluidly with their surroundings when, for example, taking pictures, recording video, or controlling remote services or devices. In various implementations, one or more components of a computing device identify an occurrence of a combined user input such as a spoken utterance and a spatial gesture made with the device at least partially coincident with the spoken utterance. The one or more components identify a content capture experience associated with the combined user input and invoke the content capture experience on the device.

Disclosed techniques enable participants of a communication session that is rendered within a mixed reality environment to change their view or perspective in the mixed reality environment. The participants may alter their view or perspective in the mixed reality environment using a data processing device displaying the communication session. The participant may interact with a data processing device displaying the communication session to cause the change in their view or perspective in the mixed reality environment. The participant interaction may include user touch and/or gestures with an associated data processing device to cause the change in the participant's view or perspective in the mixed reality environment. The participant may use a plurality of fingers on a display of the data processing device to zoom, pan or rotate their view or perspective in the mixed reality environment. Also, the participant may move the data processing device to cause their view or perspective in the mixed reality environment to change.

Remote view manipulation in communication session

In a device including a processor and a memory in communication with the processor is described, the memory includes executable instructions that, when executed by the processor, cause the processor to control the device to perform functions of: generating, based on a plurality of local 3D models, a global 3D model representing a portion of a real-world environment; determining a location of a 3D virtual object in the global 3D model; and generating augmentation data for rendering the 3D virtual object to be seen at a location of the real-world environment corresponding to the location of the 3D virtual object in the global 3D model.

Augmented reality

A user device comprises a network interface, a rendering module, and a scene modification module. The network interface is configured to receive a video signal from another device via a network. The rendering module is configured to control display apparatus of the user device to display a virtual element to a user of the user device, the virtual element comprising a video image derived from the video signal. The modification module is configured to generate rendering data for displaying a modified version of the virtual element at the other device. The modified version does not include said video image. The network interface is configured to transmit the rendering data to the other device via the network. Alternatively or in addition, the rendering data can be modified at the other device to the same end.

Communication system

An augmented reality (AR) system receives a plurality of local 3D models of a part of a real-world environment, each having been generated by a different AR device when located in the real-world environment. The local 3D models are combined to generate a global 3D model, at least part of which is transmitted to a device remote from the real-world environment. The global 3D model represents a greater portion of the real-environment than any of the local 3D models individually. The AR system receives rendering data from the remote device, and transmits it to an AR device when the AR device is located in the real-world environment. Alternatively, the rendering data may be transmitted from the remote device to the AR device via a network directly. The rendering data is for use in rendering a virtual object at the AR device in the real-world environment.

A user device has a display, a user interface and a network interface. The network interface receive a video data stream from another user interface. A user of the user device is able to pause the video stream. The paused image of the video stream and the video stream are both displayed on the display. The user is able to annotate the paused image.

Growing community of inventors

Seattle, WA, United States of America

Austin Seungmin Lee