Microsoft Technology Licensing, LLC

Meta Platforms, Inc.

Anis Ahmad

Daniel Gilbert Kennett

Jonathan Ross Hoof

Beibei Cheng

Benjamin William Walker

Ali Israr

Andrew C Flavell

Andrew Doxon

Nitzan Bartov

Forrest Walker Smith

Ben Walker

Jeffrey Skaanland

Patrick Christopher Sebring

Juha Pekka Kuikka

Sarah Sykes

Justin Clark

Daylon Curtis Walden

Jessica Hartcher-O′Brien

Methods and systems using a haptics library to determine whether to provide predefined haptic responses or parametrically-defined haptic responses are disclosed. A method includes receiving a desired semantic attribute for a haptic response to be provided while a user is using an application that is executing on an electronic device. In accordance with a determination, made by the haptics library that is distinct from the application and that is configured to provide data used to generate haptic responses to applications executing on the electronic device, that the electronic device should use a parametrically-defined haptic response to deliver a haptic response having the desired semantic attribute, selecting, by the haptics library, respective values for a set of haptic-response parameters; and providing instructions to cause haptic generators of the electronic device to deliver a parametrically-defined haptic response having the desired semantic attribute using the respective values for the set of haptic-response parameters.

Using a haptic effects library to determine whether to provide predefined or parametrically-defined haptic responses, and systems and methods of use thereof

A facial skin mask may be generated based on isolating a head part in a captured image, removing a first pixel that is indicative of non-skin from the head part in the captured image, and removing a second pixel that is indicative of having a high velocity from the head part in the captured image. Heart rate may be detected based on the change of color of the pixels of the generated facial skin mask.

Facial skin mask generation for heart rate detection

Heartrate tracking is performed entirely optically without the subject being required to wear any monitoring equipment by processing a combination of signals representing frames of video of the sinusoidal motion of a subject's facial skin color changes captured by both IR and visible light (e.g., RGB&#x2014;red/green/blue) cameras. The IR and RGB graphs that result from the processing are perfectly phase-shifted so that when the IR signal is going down in amplitude, the RGB signal is going up. Such phase-shifting enables the optical heartrate tracking to utilize diverse input feeds so that a tracked signal is accepted as the user's true heartrate when both IR and RGB signals are well correlated.

Optical heartrate tracking

Embodiments described herein use depth images to extract user behavior, wherein each depth image specifies that a plurality of pixels correspond to a user. In certain embodiments, one or more average extremity positions of a user, which can also be referred to as average positions of extremity blobs, are extracted from a depth image. An application is then updated based on the average positions of extremity blobs.

Extraction of user behavior from depth images

Embodiments described herein can be used to detect holes in a subset of pixels of a depth image that has been specified as corresponding to a user, and to fill such detected holes. Additionally, embodiments described herein can be used to produce a low resolution version of a subset of pixels that has been specified as corresponding to a user, so that when an image including a representation of the user is displayed, the image respects the shape of the user, yet is not a mirror image of the user. Further, embodiments described herein can be used to identify pixels, of a subset of pixels specified as corresponding to the user, that likely correspond to a floor supporting the user. This enables the removal of the pixels, identified as likely corresponding to the floor, from the subset of pixels specified as corresponding to the user.

Depth image processing

Embodiments described herein use depth images to extract user behavior, wherein each depth image specifies that a plurality of pixels correspond to a user. In certain embodiments, information indicative of an angle and/or curvature of a user's body is extracted from a depth image. This can be accomplished by fitting a curve to a portion of a plurality of pixels (of the depth image) that correspond to the user, and determining the information indicative of the angle and/or curvature of the user's body based on the fitted curve. An application is then updated based on the information indicative of the angle and/or curvature of the user's body. In certain embodiments, one or more average extremity positions of a user, which can also be referred to as average positions of extremity blobs, are extracted from a depth image. An application is then updated based on the average positions of extremity blobs.

Techniques described herein use signal analysis to detect and analyze repetitive user motion that is captured in a 3D image. The repetitive motion could be the user exercising. One embodiment includes analyzing image data that tracks a user performing a repetitive motion to determine data points for a parameter that is associated with the repetitive motion. The different data points are for different points in time. A parameter signal of the parameter versus time that tracks the repetitive motion is formed. The parameter signal is divided into brackets that delineate one repetition of the repetitive motion from other repetitions of the repetitive motion. A repetition in the parameter signal is analyzed using a signal processing technique. Curve fitting and/or autocorrelation may be used to analyze the repetition.

Signal analysis for repetition detection and analysis

User body angle, curvature and average extremity positions extraction using depth images

Embodiments described herein use depth images to extract user behavior, wherein each depth image specifies that a plurality of pixels correspond to a user. A depth-based center-of-mass position is determined for the plurality of pixels that correspond to the user. Additionally, a depth-based inertia tensor can also be determined for the plurality of pixels that correspond to the user. In certain embodiments, the plurality of pixels that correspond to the user are divided into quadrants and a depth-based quadrant center-of-mass position is determined for each of the quadrants. Additionally, a depth-based quadrant inertia tensor can be determined for each of the quadrants. Based on one or more of the depth-based center-of-mass position, the depth-based inertial tensor, the depth-based quadrant center-of-mass positions or the depth-based quadrant inertia tensors, an application is updated.

Growing community of inventors

Kirkland, WA, United States of America

Anis Ahmad