Microsoft Technology Licensing, LLC

Kyungsuk David Lee

Alexandru Octavian Balan

Mark James Finocchio

Stephen G Latta

Kevin Geisner

Daniel Joseph McCulloch

Brian James Mount

Jeffrey Neil Margolis

Relja Markovic

Ben J Sugden

Shawn Crispin Wright

Anthony James Ambrus

Nahil Tawfik Sharkasi

Richard E Moore

Daniel Lee Osborn

David C Haley, Jr

Albert Robles

Dave Hill

Andrew R Campbell

Christopher Douglas Edmonds

Adam James Muff

Rod G Fleck

Kathryn Stone Perez

Alex Aben-Athar Kipman

John Allen Tardif

Arthur Charles Tomlin

Jason Bradley Scott

Robert L Crocco, Jr

Jonathan Steed

Susan Elizabeth Carrie

Gregory Nelson Snook

Sheridan Martin Small

Britta Silke Hummel

Roger Sebastian Kevin Sylvan

Oscar Omar Garza Santos

Mark Mihelich

Arnulfo Zepeda Navratil

Randy Crane

Jamie D J Shotton

David Jason Christopher Horbach

Matthew Jon Puls

Two-handed interactions with a natural user interface are disclosed. For example, one embodiment provides a method comprising detecting via image data received by the computing device a context-setting input performed by a first hand of a user, and sending to a display a user interface positioned based on a virtual interaction coordinate system, the virtual coordinate system being positioned based upon a position of the first hand of the user. The method further includes detecting via image data received by the computing device an action input performed by a second hand of the user, the action input performed while the first hand of the user is performing the context-setting input, and sending to the display a response based on the context-setting input and an interaction between the action input and the virtual interaction coordinate system.

Two-hand interaction with natural user interface

Embodiments are disclosed that relate to processing image pixels. For example, one disclosed embodiment provides a system for classifying pixels comprising retrieval logic; a pixel storage allocation including a plurality of pixel slots, each pixel slot being associated individually with a pixel, where the retrieval logic is configured to cause the pixels to be allocated into the pixel slots in an input sequence; pipelined processing logic configured to output, for each of the pixels, classification information associated with the pixel; and scheduling logic configured to control dispatches from the pixel slots to the pipelined processing logic, where the scheduling logic and pipelined processing logic are configured to act in concert to generate the classification information for the pixels in an output sequence that differs from and is independent of the input sequence.

System and method for classifying pixels

An augmented reality submission includes a hologram to virtually augment a world space object and a compensation offer for presenting the hologram to a viewer of the world space object. The augmented reality submission is selected as a winning submission if the submission satisfies a selection criteria.

Augmented reality auction platform

A depth sensor obtains images of articulated portions of a user's body such as the hand. A predefined model of the articulated body portions is provided. Representative attract points of the model are matched to centroids of the depth sensor data, and a rigid transform of the model is performed, in an initial, relatively coarse matching process. This matching process is then refined in a non-rigid transform of the model, using attract point-to-centroid matching. In a further refinement, an iterative process rasterizes the model to provide depth pixels of the model, and compares the depth pixels of the model to the depth pixels of the depth sensor. The refinement is guided by whether the depth pixels of the model are overlapping or non-overlapping with the depth pixels of the depth sensor. Collision, distance and angle constraints are also imposed on the model.

Probabilistic and constraint based articulated model fitting

Technology is described for displaying a collision between objects by an augmented reality display device system. A collision between a real object and a virtual object is identified based on three dimensional space position data of the objects. At least one effect on at least one physical property of the real object is determined based on physical properties of the real object, like a change in surface shape, and physical interaction characteristics of the collision. Simulation image data is generated and displayed simulating the effect on the real object by the augmented reality display. Virtual objects under control of different executing applications can also interact with one another in collisions.

Displaying a collision between real and virtual objects

Embodiments related to detecting object information from image data collected by an image sensor are disclosed. In one example embodiment, the object information is detected by receiving a frame of image data from the image sensor and detecting a change in a threshold condition related to an object within the frame. The embodiment further comprises adjusting a setting that changes a power consumption of the image sensor in response to detecting the threshold condition.

Controlling power consumption in object tracking pipeline

Technology is described for providing realistic occlusion between a virtual object displayed by a head mounted, augmented reality display system and a real object visible to the user's eyes through the display. A spatial occlusion in a user field of view of the display is typically a three dimensional occlusion determined based on a three dimensional space mapping of real and virtual objects. An occlusion interface between a real object and a virtual object can be modeled at a level of detail determined based on criteria such as distance within the field of view, display size or position with respect to a point of gaze. Technology is also described for providing three dimensional audio occlusion based on an occlusion between a real object and a virtual object in the user environment.

Realistic occlusion for a head mounted augmented reality display

A planar surface within a physical environment is detected enabling presentation of a graphical user interface overlaying the planar surface. Detection of planar surfaces may be performed, in one example, by obtaining a collection of three-dimensional surface points of a physical environment imaged via an optical sensor subsystem. A plurality of polygon sets of points are sampled within the collection. Each polygon set of points includes three or more localized points of the collection that defines a polygon. Each polygon is classified into one or more groups of polygons having a shared planar characteristic with each other polygon of that group. One or more planar surfaces within the collection are identified such that each planar surface is at least partially defined by a group of polygons containing at least a threshold number of polygons.

Planar surface detection

Automatic depth camera aiming is provided by a method which includes receiving from the depth camera one or more observed depth images of a scene. The method further includes, if a point of interest of a target is found within the scene, determining if the point of interest is within a far range relative to the depth camera. The method further includes, if the point of interest of the target is within the far range, operating the depth camera with a far logic, or if the point of interest of the target is not within the far range, operating the depth camera with a near logic.

Automatic depth camera aiming

A system and method are disclosed relating to a pipeline for generating a computer model of a target user, including a hand model of the user's hands, captured by an image sensor in a NUI system. The computer model represents a best estimate of the position of a user's hand or hands and whether the hand or hand is in an open or closed state. The generated hand model may be used by a gaming or other application to determine such things as user gestures and control actions.

System for recognizing an open or closed hand

A system and method are disclosed relating to a pipeline for generating a computer model of a target user, including a hand model of the user's hands and fingers, captured by an image sensor in a NUI system. The computer model represents a best estimate of the position and orientation of a user's hand or hands. The generated hand model may be used by a gaming or other application to determine such things as user gestures and control actions.

System for finger recognition and tracking

Technology is described for determining and using invariant features for computer vision. A local orientation may be determined for each depth pixel in a subset of the depth pixels in a depth map. The local orientation may an in-plane orientation, an out-out-plane orientation or both. A local coordinate system is determined for each of the depth pixels in the subset based on the local orientation of the corresponding depth pixel. A feature region is defined relative to the local coordinate system for each of the depth pixels in the subset. The feature region for each of the depth pixels in the subset is transformed from the local coordinate system to an image coordinate system of the depth map. The transformed feature regions are used to process the depth map.

Invariant features for computer vision

Systems, methods and computer readable media are disclosed for a gesture tool. A capture device captures user movement and provides corresponding data to a gesture recognizer engine and an application. From that, the data is parsed to determine whether it satisfies one or more gesture filters, each filter corresponding to user-performed gesture. The data and the information about the filters is also sent to a gesture tool, which displays aspects of the data and filters. In response to user input corresponding to a change in a filter, the gesture tool sends an indication of such to the gesture recognizer engine and application, where that change occurs.

Gesture tool

A depth sensor obtains images of articulated portions of a user's body such as the hand. A predefined model of the articulated body portions is provided. The model is matched to corresponding depth pixels which are obtained from the depth sensor, to provide an initial match. The initial match is then refined using distance constraints, collision constraints, angle constraints and a pixel comparison using a rasterized model. Distance constraints include constraints on distances between the articulated portions of the hand. Collision constraints can be enforced when the model meets specified conditions, such as when at least two adjacent finger segments of the model are determined to be in a specified relative position, e.g., parallel. The rasterized model includes depth pixels of the model which are compared to identify overlapping pixels. Dimension of the articulated portions of the model are individually adjusted.

Fully automatic dynamic articulated model calibration

Growing community of inventors

Redmond, WA, United States of America

Kyungsuk David Lee