Microsoft Technology Licensing, LLC

Jonathan Campbell

Harish Kulkarni

Peter John Ansell

Akhilesh Kaza

Zhenguang Chen

Charles Reeves Little, Ii

Boxin Li

Arun Keller Krishnamoorthy

Charles Keller Smith

Jatin Sharma

Brian William Timm

Andrew Joseph Violino

Eric Norman Badger

Viktoriya Taranov

Jay Beavers

Jacob Daniel Cohen

Dmytro Rudchenko

Fei Su

Austin Bradley Hodges

Ann M Paradiso

Narasimhan Raghunath

Eray Chou

Meredith June Morris

Aleksandar Uzelac

Alexander H Malek

Edward Bryan Cutrell

Todd Haugen

Vishwac Sena Kannan

Daniel J Hwang

Peter Kenneth Harwood

Constantin Stanciu

Mohammed Nazeeruddin

Anthony Lawrence Jackson

Howard Morrison Crow, Iii

Bradley Stevenson

Ping Jiang

Alexander John Fiannaca

Kerem Yuceturk

Howard M Crow, Iii

Christopher Foster Johnson

Kiley Rebecca Sobel

Dwayne Lamb

Mark E Phair

Sean K Gabriel

Dhivya Padmanabhan

Robin Marie Miller

Peter K Harwood

Howard M Iii Crow

Systems and methods are provided for collecting eye-gaze data for training an eye-gaze prediction model. The collecting includes selecting a scan path passing through a series of regions of a grid on a screen of a computing device, moving a symbol as an eye-gaze target along the scan path, and receiving facial images at eye-gaze points. The eye-gaze points are uniformly distributed within the respective regions. Areas of the regions that are adjacent to edges and corners of the screen are smaller than other regions. The difference in areas shifts centers of the regions toward the edges, density of data closer to the edges. The scan path passes through locations in proximity to the edges and corners of the screen for capturing more eye-gaze points in the proximity. The methods interactively enhance variations of facial images by displaying instructions to the user to make specific actions associated with the face.

Systems and methods of capturing eye-gaze data

Systems and methods are provided for predicting an eye gaze location of an operator of a computing device. In particular, the method generates an image grid that includes regions of interest based on a facial image. The facial image is based on a received image frame of a video stream that captures the operator using the computing device. The image grid further includes a region that indicate rotation information of the face. The method further uses a combination of trained neural networks to extract features of the regions of interest in the image grid and predict the eye gaze location on the screen of the computing device. The trained set of neural networks includes a convolutional neural network. The method optionally generate head pose pitch, roll, and yaw information to improve accuracy of predicting the location of an eye gaze.

Systems and methods for gaze-tracking

Systems and methods are provided for collecting eye-gaze data for training an eye-gaze prediction model. The collecting includes selecting a scan path that passes through a series of regions of a grid on a screen of a computing device, moving a symbol as an eye-gaze target along the scan path, and receiving facial images at eye-gaze points. The eye-gaze points are uniformly distributed within the respective regions. Areas of the regions that are adjacent to edges and corners of the screen are smaller than other regions. The difference in areas shifts centers of the regions toward the edges, density of data closer to the edges. The scan path passes through locations in proximity to the edges and corners of the screen for capturing more eye-gaze points in the proximity. The methods interactively enhance variations of facial images by displaying instructions to the user to make specific actions associated with the face.

Systems and methods disclosed herein are related to an intelligent UI element selection system using eye-gaze technology. In some example aspects, a UI element selection zone may be determined. The selection zone may be defined as an area surrounding a boundary of the UI element. Gaze input may be received and the gaze input may be compared with the selection zone to determine an intent of the user. The gaze input may comprise one or more gaze locations. Each gaze location may be assigned a value according to its proximity to the UI element and/or its relation to the UI element's selection zone. Each UI element may be assigned a threshold. If the aggregated value of gaze input is equal to or greater than the threshold for the UI element, then the UI element may be selected.

Intelligent user interface element selection using eye-gaze

Systems and methods disclosed herein relate to assigning dynamic eye-gaze dwell-times. Dynamic dwell-times may be tailored to the individual user. For example, a dynamic dwell-time system may be configured to receive data from the user, such as the duration of time the user takes to execute certain actions within applications (e.g., read a word suggestion before actually selecting it). The dynamic dwell-time system may also prevent users from making unintended selections by providing different dwell times for different buttons. Specifically, on a user interface, longer dwell times may be established for the critical keys (e.g., 'close' program key, “send” key, word suggestions, and the like) and shorter dwell times may be established for the less critical keys (e.g., individual character keys on a virtual keyboard, spacebar, backspace, and the like).

Dynamic eye-gaze dwell times

Generation of expressive content is provided. An expressive synthesized speech system provides improved voice authoring user interfaces by which a user is enabled to efficiently author content for generating expressive output. An expressive synthesized speech system provides an expressive keyboard for enabling input of textual content and for selecting expressive operators, such as emoji objects or punctuation objects for applying predetermined prosody attributes or visual effects to the textual content. A voicesetting editor mode enables the user to author and adjust particular prosody attributes associated with the content for composing carefully-crafted synthetic speech. An active listening mode (ALM) is provided, which when selected, a set of ALM effect options are displayed, wherein each option is associated with a particular sound effect and/or visual effect. The user is enabled to rapidly respond with expressive vocal sound effects or visual effects while listening to others speak.

User interface for generating expressive content

Systems and methods disclosed herein relate to assigning dynamic eye-gaze dwell-times. Dynamic dwell-times may be tailored to the individual user. For example, a dynamic dwell-time system may be configured to receive data from the user, such as the duration of time the user takes to execute certain actions within applications (e.g., read a word suggestion before actually selecting it). The dynamic dwell-time system may also prevent users from making unintended selections by providing different dwell times for different buttons. Specifically, on a user interface, longer dwell times may be established for the critical keys (e.g., 'close' program key, 'send' key, word suggestions, and the like) and shorter dwell times may be established for the less critical keys (e.g., individual character keys on a virtual keyboard, spacebar, backspace, and the like).

A user's signature may be decomposed into one or more components. The components may be described using one or more control points. A user may sign with their eyes by focusing their gaze on a set of these control points that make up the signature. If the user's gaze is within a threshold distance from a control point the signature is validated. Moreover, by modifying the control points based upon the actual gaze location (which is within the threshold distance), the signature may be slightly modified. For example, the signature may be decomposed into one or more components as Bezier curves and the user may be asked to focus on each control point of each of the one or more Bezier curves. Modifying the control points of a Bezier curve slightly still produces a smooth curve, but introduces natural variations.

Signing with your eyes

A set of user interfaces is provided to an administrator of a website. The user interfaces enable the administrator to configure search functionality of the website to present different types of search results differently. For instance, the administrator can use the user interfaces to configure the search functionality to present different properties for search results belonging to different result types.

Result types for conditional data display

The systems and methods described herein assist persons with the use of computers based on eye gaze, and allow such persons to control such computing systems using various eye trackers. The systems and methods described herein use eye trackers to control cursor (or some other indicator) positioning on an operating system using the gaze location reported by the eye tracker. The systems and methods described herein utilize an interaction model that allows control of a computer using eye gaze and dwell. The data from eye trackers provides a gaze location on the screen. The systems and methods described herein control a graphical user interface that is part of an operating system relative to cursor positioning and associated actions such as Left-Click, Right-Click, Double-Click, and the like. The interaction model presents appropriate user interfaces to navigate the user through applications on the computing system.

Controlling a computer using eyegaze and dwell

A client device presents a resource to a user. In order to present the resource, the client device generates and renders one or more presentation strings. The client device generates a presentation string by identifying a template module that corresponds to an content resource object in a dataset downloaded by the client device. The client device then performs a template execution operation on the content resource object. The template module corresponding to the content resource object specifies the template execution operation.

Templates for displaying data

Techniques are described for discovering capabilities of voice-enabled resources. A voice-controlled digital personal assistant can respond to user requests to list available voice-enabled resources that are capable of performing a specific task using voice input. The voice-controlled digital personal assistant can also respond to user requests to list the tasks that a particular voice-enabled resource can perform using voice input. The voice-controlled digital personal assistant can also support a practice mode in which users practice voice commands for performing tasks supported by voice-enabled resources.

Discovering capabilities of third-party voice-enabled resources

Growing community of inventors

Redmond, WA, United States of America

Jonathan Campbell