Microsoft Technology Licensing, LLC

Affectiva, Inc.

Massachusetts Institute of Technology

Daniel J McDuff

Rana El Kaliouby

Javier Hernandez Rivera

Seyedmohammad Mavadati

Erroll William Wood

Tadas Baltrusaitis

Rosalind Picard

Mary P Czerwinski

Panu James Turcot

Abdelrahman N Mahmoud

Jin A Suh

Kael R Rowan

Prasanth Murali

Sarah Pratt

Mohammad Akram

Weixuan Chen

May Amr Fouad

Sarah Gontarek

Nicholas Langeveld

Systems and methods are provided that are directed to generating video sequences including physio-realistic avatars. In examples, an albedo for an avatar is received, a sub-surface skin color associated with the albedo is modified based on physiological data associated with physiologic characteristic, and an avatar based on the albedo and the modified sub-surface skin color is rendered. The rendered avatar may then be synthesized in a frame of video. In some examples, a video including the synthesized avatar may be used to train a machine learning model to detect a physiological characteristic. The machine learning model may receive a plurality of video segments, where one or more of the video segments includes a synthetic physio-realistic avatar generated with the physiological characteristic. The machine learning model may be trained using the plurality of video segments. The trained model may be provided to a requesting entity.

Generating physio-realistic avatars for training non-contact models to recover physiological characteristics

The present disclosure relate to highlighting audience members with reactions to a presenter of an online meeting. Unlike physical, fact-to-face meeting that enables spontaneous interactions among the presenter and the audiences that are collocated with the presenter, presenting materials during an online meeting raises an issue of the present not being able to see real-time reactions or feedback by the audience members. The present disclosure addresses the issue by dynamically determining one or more audience members who indicate reactions during the online meeting or presentation and displaying faces of the one or more audience members under spotlight to the presenter. The presenter sees faces of the audience members with reactions during the online presentation and responds to the audience members and keep the audience engaged. The spotlight audience server analyzes video frames and determines types of reactions of the audience members.

Highlighting expressive participants in an online meeting

Systems and methods are directed to animating subtle physiological processes directly on avatars and photos. That is, physiologically-grounded spatial, color space, and temporal modifications may be made to the appearance of an avatar to simulate a physiological characteristic, such as blood flow. More specifically, a frame of a video sequence and a physiological signal may be received. An attention mask may be generated based on the received physiological signal, where the attention mask includes attention weights indicative of a strength of the physiological signal for differing portions of the frame of the video sequence. Accordingly, a pixel adjustment value based on the physiological signal and the attention mask may be generated and applied to an identified pixel in the frame of the video sequence.

Animating physiological characteristics on 2D or 3D avatars

Disclosed embodiments provide for deep convolutional neural network computing. The convolutional computing is accomplished using a multilayered analysis engine. The multilayered analysis engine includes a deep learning network using a convolutional neural network (CNN). The multilayered analysis engine is used to analyze multiple images in a supervised or unsupervised learning process. Multiple images are provided to the multilayered analysis engine, and the multilayered analysis engine is trained with those images. A subject image is then evaluated by the multilayered analysis engine. The evaluation is accomplished by analyzing pixels within the subject image to identify a facial portion and identifying a facial expression based on the facial portion. The results of the evaluation are output. The multilayered analysis engine is retrained using a second plurality of images.

Convolutional computing using multilayered analysis engine

Techniques for performing viewing verification using a plurality of classifiers are disclosed. Images of an individual may be obtained concurrently with an electronic display presenting one or more images. Image classifiers for facial and head pose analysis may be obtained. The images of the individual may be analyzed to identify a face of the individual in one of the plurality of images. A viewing verification metric may be calculated using the image classifiers and a verified viewing duration of the screen images by the individual may be calculated based on the plurality of images and the analyzing. Viewing verification can involve determining whether the individual is in front of the screen, facing the screen, and gazing at the screen. A viewing verification metric can be generated in order to determine a level of interest of the individual in particular media and images.

Electronic display viewing verification

A machine learning system is accessed. The machine learning system is used to translate content into a representative icon. The machine learning system is used to manipulate emoji. The machine learning system is used to process an image of an individual. The machine learning processing includes identifying a face of the individual. The machine learning processing includes classifying the face to determine facial content using a plurality of image classifiers. The classifying includes generating confidence values for a plurality of action units for the face. The facial content is translated into a representative icon. The translating the facial content includes summing the confidence values for the plurality of action units. The representative icon comprises an emoji. A set of emoji can be imported. The representative icon is selected from the set of emoji. The emoji selection is based on emotion content analysis of the face.

Emoji manipulation using machine learning

Images are analyzed using sub-sectional component evaluation in order to augment classifier usage. An image of an individual is obtained. The face of the individual is identified, and regions within the face are determined. The individual is evaluated to be within a sub-sectional component of a population based on a demographic or based on an activity. An evaluation of content of the face is performed based on the individual being within a sub-sectional component of a population. The sub-sectional component of a population is used for disambiguating among content types for the content of the face. A Bayesian framework that includes a conditional probability is used to perform the evaluation of the content of the face, and the evaluation is further based on a prior event that occurred.

Image analysis using sub-sectional component evaluation to augment classifier usage

In illustrative implementations of this invention, a photoplethysmographic device measures variations of light that is reflected from, or transmitted through, human skin. In some implementations, the device includes a camera that takes the measurements remotely. In others, the device touches the skin during the measurements. The device includes a camera or other light sensor, which includes at least orange, green and cyan color channels. In some cases, such as a contact device, the device includes three or more colors of active light sources, including at least orange, green and cyan light sources. A computer analyzes the sensor data, in order to estimate a cardiac blood volume pulse wave. For each cardiac pulse, a computer detects the systolic peak and diastolic inflection of the wave, by calculating a second derivative of the wave. From the estimated wave, a computer estimates heart rate, heart rate variability and respiration rate.

Methods and apparatus for physiological measurement using color band photoplethysmographic sensor

Frames of a video frame sequence capturing one or more skin regions of a body are provided to a first neural network. The first neural network generates respective appearance representations based on the frames. An appearance representation generated based on a particular frame is indicative of a spatial distribution of a physiological signal across the particular frame. Simultaneously with providing the frames to the first neural network, the frames are also provided to a second neural network. The second neural network determines the physiological signal based on the frames. Determining the physiological signal by the second neural network includes applying the appearance representations, generated by the first neural network, to outputs of one or more layers of the second neural network to emphasize regions, in the frames, that exhibit relatively stronger presence of the physiological signal and deemphasize regions, in the frames, that exhibit relatively weaker presence of physiological signal.

Growing community of inventors

Cambridge, MA, United States of America

Daniel J McDuff