Microsoft Technology Licensing, LLC

Peter John Ansell

Jonathan Campbell

Harish Kulkarni

Hrvoje Benko

David R Perek

Eric Norman Badger

Christopher Harry Stoumbos

Akhilesh Kaza

John Lawrence Miller

Jacob Daniel Cohen

Robert W Young

Oyvind Haehre

Shahram Izadi

Dmytro Rudchenko

Jay Beavers

Fei Su

Austin Bradley Hodges

Ann M Paradiso

Jatin Sharma

Narasimhan Raghunath

Dwayne Lamb

Systems and methods are provided for predicting an eye gaze location of an operator of a computing device. In particular, the method generates an image grid that includes regions of interest based on a facial image. The facial image is based on a received image frame of a video stream that captures the operator using the computing device. The image grid further includes a region that indicate rotation information of the face. The method further uses a combination of trained neural networks to extract features of the regions of interest in the image grid and predict the eye gaze location on the screen of the computing device. The trained set of neural networks includes a convolutional neural network. The method optionally generate head pose pitch, roll, and yaw information to improve accuracy of predicting the location of an eye gaze.

Systems and methods for gaze-tracking

Systems and methods disclosed herein relate to assigning dynamic eye-gaze dwell-times. Dynamic dwell-times may be tailored to the individual user. For example, a dynamic dwell-time system may be configured to receive data from the user, such as the duration of time the user takes to execute certain actions within applications (e.g., read a word suggestion before actually selecting it). The dynamic dwell-time system may also prevent users from making unintended selections by providing different dwell times for different buttons. Specifically, on a user interface, longer dwell times may be established for the critical keys (e.g., 'close' program key, 'send' key, word suggestions, and the like) and shorter dwell times may be established for the less critical keys (e.g., individual character keys on a virtual keyboard, spacebar, backspace, and the like).

Dynamic eye-gaze dwell times

Systems and methods disclosed herein are related to an intelligent UI element selection system using eye-gaze technology. In some example aspects, a UI element selection zone may be determined. The selection zone may be defined as an area surrounding a boundary of the UI element. Gaze input may be received and the gaze input may be compared with the selection zone to determine an intent of the user. The gaze input may comprise one or more gaze locations. Each gaze location may be assigned a value according to its proximity to the UI element and/or its relation to the UI element's selection zone. Each UI element may be assigned a threshold. If the aggregated value of gaze input is equal to or greater than the threshold for the UI element, then the UI element may be selected.

Intelligent user interface element selection using eye-gaze

The systems and methods described herein assist persons with the use of computers based on eye gaze, and allow such persons to control such computing systems using various eye trackers. The systems and methods described herein use eye trackers to control cursor (or some other indicator) positioning on an operating system using the gaze location reported by the eye tracker. The systems and methods described herein utilize an interaction model that allows control of a computer using eye gaze and dwell. The data from eye trackers provides a gaze location on the screen. The systems and methods described herein control a graphical user interface that is part of an operating system relative to cursor positioning and associated actions such as Left-Click, Right-Click, Double-Click, and the like. The interaction model presents appropriate user interfaces to navigate the user through applications on the computing system.

Controlling a computer using eyegaze and dwell

Transferring data via audio link is described. In an example a short sequence of data can be transferred between two devices by encoding the sequence of data as an audio sequence. For example, the audio sequence may be a sequence of tones which vary in dependence on the encoded data. The sequence of data may be encoded by a first device and transmitted using a loudspeaker associated with the first device. At least one mobile communications device can be used to capture the audio sequence, for example using a microphone, and to decode the sequence, retrieving the data encoded therein. In some examples the encoded data may comprise a shortened URL or other information which can be used to control one or more aspects of the capture device.

Transferring data via audio link

Gesture recognition is described. In one example, gestures performed by a user of an input device having a touch-sensitive portion are detected using a definition of a number of regions corresponding to zones on the touch-sensitive portion, each region being associated with a distinct set of gestures. Data describing movement of the user's digits on the touch-sensitive portion is received, and an associated region for the data determined. The data is compared to the associated region's set of gestures, and a gesture applicable to the data selected. A command associated with the selected gesture can then be executed. In an example, comparing the data to the set of gestures comprises positioning a threshold for each gesture relative to the start of the digit's movement. The digit's location is compared to each threshold to determine whether a threshold has been crossed, and, if so, selecting the gesture associated with that threshold.

Gesture recognition

An input device has both a touch sensor and a position sensor. A computer using data from the input device uses the relative motion of a contact on a touch sensor with respect to motion from a position detector to disambiguate intentional from incidental motion. The input device provides synchronized position sensor and touch sensor data to the computer to permit processing the relative motion and performing other computations on both position sensor and touch sensor data. The input device can encode the magnitude and direction of motion of the position sensor and combines it with the touch sensor data from the same time frame, and output the synchronized data to the computer.

Disambiguating intentional and incidental contact and motion in multi-touch pointing devices

Growing community of inventors

Watford, United Kingdom

Peter John Ansell