Microsoft Technology Licensing, LLC

Szymon Piotr Stachniak

Tommer Leyvand

Craig C Peeper

Shao Liu

Johnny Hsienchow Lee

Johnny Chung Lee

Kenneth Alan Lobb

Yuri Pekelny

Charles Claudius Marais

Joseph C Bertolami

Zhengyou Zhang

Baining Guo

Cyrus Kanga

Mark D Schwesinger

Kathleen Patricia Mulcahy

Shiraz Jitendra Cupala

Zhangwei Xu

Mario Esposito

Zsolt Mathe

Robert Thomas Held

Prafulla Jinendra Masalkar

Anatolie Gavriliuc

Stanley William Adermann

Andrew Frederick Muehlhausen

Jason Robert Tuck

Neil Warren Black

Leonardo G Del Castillo

Michael C Johnson

Zhuowen Tu

Baoyuan Wang

Mark Plagge

Russell P Songco

Riccardo Giraldi

Ke Deng

Maxime Ouellet

William Douglas Guyman

William F Schiefelbein

Clinton Chi-Wen Woon

Marc Pollefeys

John Darren Elliott

Michelle Brook

Nazeeh Amin Eldirghami

Spencer Dean Perreault

Joshua C Zana

Hendrik Mark Langerak

Parham Mohadjer

Howard Benjamin Nager

Jon D Pulsipher

Ali Osman Ulusoy

Patrick Orville Cook

Michael Taylor Janney

Kien Toan Huynh

Robert John Fitzgerald

Ronald Forbes

Muhammad Jabir Kapasi

Alexander D Tudor

Heather Grantham Brodie

James Chadon Foster

Wayne Richard Bisson

David C Kline

Ronald Omega Forbes, Jr

Silvano Galliani

Matthew Alan Medak

Rahul Sawhney

Eric Christopher Voreis

Ronald Oswin Pessner Jr

Marc Victor Haddah

Alexander James Soule

Leonardo Del Castillo

Embodiments of the present invention analyze a context in which a user interacts with a computer interface and automatically optimizes the interface for the context. The controller or control mode the user selects for interaction may define the context, in part. Examples of control modes include gesturing, audio control, use of companion devices, and use of dedicated control devices, such as game controllers and remote controls. The different input devices are designed for different tasks. Nevertheless, a user will frequently attempt to perform a task using a control input that is not adapted for the task. Embodiments of the present invention change the characteristics of the user interface to make it easier for the user to complete an intended task using the input device of the user's choice.

Contextual user interface

The described implementations relate to images and depth information and generating useful information from the images and depth information. One example can identify planes in a semantically-labeled 3D voxel representation of a scene. The example can infer missing information by extending planes associated with structural elements of the scene. The example can also generate a watertight manifold representation of the scene at least in part from the inferred missing information.

Deriving information from images

Improved techniques for performing object segmentation are disclosed. Surface reconstruction (SR) data corresponding to an environment is accessed. This SR data is used to generate a detailed three-dimensional (3D) representation of the environment. The SR data is also used to infer a high-level 3D structural representation of the environment. The high-level 3D structural representation is inferred using machine learning that is performed on the surface reconstruction data to identify a structure of the environment. The high-level 3D structural representation is then cut from the detailed 3D representation. This cutting process generates a clutter mesh comprising objects that remain after the cut and that are distinct from the structure. Object segmentation is then performed on the remaining objects to identify those objects.

Object segmentation using high-level structural meshes

A computer system is provided that includes a camera device and a processor configured to receive scene data captured by the camera device for a three-dimensional environment that includes one or more physical objects, generate a geometric representation of the scene data, process the scene data using an artificial intelligence machine learning model that outputs object boundary data and object labels, augment the geometric representation with the object boundary data and the object labels, and identify the one or more physical objects based on the augmented geometric representation of the three-dimensional environment. For each identified physical object, the processor is configured to generate an associated virtual object that is fit to one or more geometric characteristics of that identified physical object. The processor is further configured to track each identified physical object and associated virtual object across successive updates to the scene data.

Object permanence in surface reconstruction

An augmented reality device includes a logic machine and a storage machine holding instructions executable by the logic machine to, for one or more real-world surfaces represented in a three-dimensional representation of a real-world environment of the augmented reality device, fit a virtual two-dimensional plane to the real-world surface. A request to place a virtual three-dimensional object on the real-world surface is received. For each of a plurality of candidate placement locations on the virtual two-dimensional plane, the candidate placement location is evaluated as a valid placement location or an invalid placement location for the virtual three-dimensional object. An invalidation mask is generated that defines the valid and invalid placement locations on the virtual two-dimensional plane.

Virtual object placement for augmented reality

A computing system is provided, including one or more optical sensors, a display, one or more user input devices, and a processor. The processor may receive optical data of a physical environment. Based on the optical data, the processor may generate a three-dimensional representation of the physical environment. For at least one target region of the physical environment, the processor may generate a three-dimensional bounding volume surrounding the target region based on a depth profile measured by the one or more optical sensors and/or estimated by the processor. The processor may generate a two-dimensional bounding shape at least in part by projecting the three-dimensional bounding volume onto an imaging surface of an optical sensor. The processor may output an image of the physical environment and the two-dimensional bounding shape for display. The processor may receive a user input and modify the two-dimensional bounding shape based on the user input.

Image bounding shape using 3D environment representation

Examples are disclosed herein that relate to calibrating a user's eye for a stereoscopic display. One example provides, on a head-mounted display device including a see-through display, a method of calibrating a stereoscopic display for a user's eyes, the method including for a first eye, receiving an indication of alignment of a user-controlled object with a first eye reference object viewable via the head-mounted display device from a perspective of the first eye, determining a first ray intersecting the user-controlled object and the first eye reference object from the perspective of the first eye, and determining a position of the first eye based on the first ray. The method further includes repeating such steps for a second eye, determining a position of the second eye based on a second ray, and calibrating the stereoscopic display based on the position of the first eye and the position of the second eye.

Calibrating a near-eye display

An image such as a depth image of a scene may be received, observed, or captured by a device. A grid of voxels may then be generated based on the depth image such that the depth image may be downsampled. A background included in the grid of voxels may then be discarded to isolate one or more voxels associated with a foreground object such as a human target and the isolated voxels associated with the foreground object may be processed.

Systems and methods for removing a background of an image

An image such as a depth image of a scene may be received, observed, or captured by a device. A grid of voxels may then be generated based on the depth image such that the depth image may be downsampled. A background included in the grid of voxels may also be removed to isolate one or more voxels associated with a foreground object such as a human target. A location or position of one or more extremities of the isolated human target may then be determined.

Methods and systems for determining and tracking extremities of a target

Low-latency gesture detection is described, for example, to compute a gesture class from a live stream of image frames of a user making a gesture, for example, as part of a natural user interface controlling a game system or other system. In examples, machine learning components are trained to learn gesture primitives and at test time, are able to detect gestures using the learned primitives, in a fast, accurate manner. For example, a gesture primitive is a latent (unobserved) variable features of a subset of frames from a sequence of frames depicting a gesture. For example, the subset of frames has many fewer frames than a sequence of frames depicting a complete gesture. In various examples gesture primitives are learnt from instance level features computed by aggregating frame level features to capture temporal structure. In examples frame level features comprise body position and body part articulation state features.

Low-latency gesture detection

An image such as a depth image of a scene may be received, observed, or captured by a device. A grid of voxels may then be generated based on the depth image such that the depth image may be downsampled. A background included in the grid of voxels may also be removed to isolate one or more voxels associated with a foreground object such as a human target. A location or position of one or more extremities of the isolated human target may be determined and a model may be adjusted based on the location or position of the one or more extremities.

Human tracking system

A method to identify a targeted object based on eye tracking and gesture recognition. The method is enacted in a compute system controlled by a user and operatively coupled to a machine vision system. In this method, the compute system receives, from the machine vision system, video imaging a head and pointer of the user. Based on the video, the compute system computes a geometric line of sight of the user, which is partly occluded by the pointer. Then, with reference to position data for one or more objects, the compute system identifies the targeted object, situated along the geometric line of sight.

Pointer projection for natural user input

Growing community of inventors

Redmond, WA, United States of America

Szymon Piotr Stachniak