Microsoft Technology Licensing, LLC

Michigan State University

Biyi Fang

Mi Zhang

Sheng Yi

Tianyi Chen

Xiao Zeng

Claes-Fredrik Urban Mannby

Yibo Sun

Jenna Hong

Gilbert Antonius

Taiwoo Park

Abhay Tanaji Doke

Gowtham Ganesan

Tra My Nguyen

Ryan Serrao

Blake Emory Williford

Apurva Sandeep Gandhi

The present disclosure describes systems and methods for implementing input-adaptive neural networks. Several approaches and systems are presented for creating flexible and computationally-efficient deep neural networks that adapt their complexity based upon input data such as video streams. The approach may use fine-grained design and an automatic approach for generating optimal network architectures based on early-exit mechanisms. A neural network modified according to such principles can be implemented in various devices including mobile vision systems such as smartphones, drones, and head-mounted augmented reality devices.

Systems and methods for implementing flexible, input-adaptive deep learning neural networks

A data processing system implements training a first convolutional neural network (CNN) to classify shapes in digital ink inputs, the first CNN being trained to receive ink stroke information as an input and to output an indication whether the ink stroke information represents a hand drawn geometric shape. The data processing system further implements compressing the first CNN to generate a second CNN capable of being operated on a resource-constrained computing device lacking sufficient computing resources to operate an instance of the first CNN and installing the second CNN on the resource-constrained computing device to process ink stroke information captured by a user interface of the computing device.

Techniques for enabling on-device shape recognition

A computer system is provided that includes one or more processors configured to receive user input for inked content to a digital canvas, and process the inked content to determine one or more writing regions. Each writing region includes recognized text and one or more document layout features associated with that writing region. The one or more processors are further configured to tokenize a target writing region of the one or more writing regions into a sequence of tokens, process the sequence of tokens of the target writing region using a task extraction subsystem that operates on tokens representing both the recognized text and the one or more document layout features of the target writing region, segment the target writing region into one or more sentence segments, and classify each of the one or more sentence segments as a task sentence or a non-task sentence.

Sequence labeling task extraction from inked content

Systems and methods are disclosed which allow mobile devices, and other resource constrained applications, to more efficiently and effectively utilize deep learning neural networks using only (or primarily) local resources. These systems and methods take the dynamics of runtime resources into account to enable resource-aware, multi-tenant on-device deep learning for artificial intelligence functions for use in tasks like mobile vision systems. The multi-capacity framework enables deep learning models to offer flexible resource-accuracy trade-offs and other similar balancing of performance and resources consumed. At runtime, various systems disclosed herein may dynamically select the optimal resource-accuracy trade-off for each deep learning model to fit the model's resource demand to the system's available runtime resources and the needs of the task being performed by the model. In doing so, systems and methods disclosed herein can efficiently utilize the limited resources in mobile systems to maximize performance of multiple concurrently running neural network-based applications.

Systems and methods for providing flexible, multi-capacity models for use of deep neural networks in mobile devices

Systems and methods for performing spatial line grouping on digital ink stokes. The system includes an electronic processor configured to access a set of hypothetical lines in an electronic document and determine a set of hypothetical line pairings. The electronic processor is also configured to determine, via a gradient boosting tree model, a merge confidence score for each hypothetical line pairing and compare a first merge confidence score with a merge threshold. The first merge confidence score is associated with a first hypothetical line and a first neighboring hypothetical line. The electronic processor is also configured to, in response to the first merge confidence score satisfying the merge threshold, merge the first hypothetical line and the first neighboring hypothetical line to form a first line grouping. The electronic processor is also configured to perform a digital ink stroke analysis on the electronic document based on the first line grouping.

Gradient boosting tree-based spatial line grouping on digital ink strokes

A data processing system for performing a semantic analysis of digital ink stroke data implements obtaining the digital ink stroke data representing handwritten text, drawings, or both; analyzing the digital ink stroke data to extract path signature feature information from the digital ink stroke data; analyzing the path signature feature information using a convolutional neural network (CNN) trained to perform a pixel-level sematic analysis of the digital ink stroke data and to output a pixel segmentation map with semantic prediction information for each pixel of digital ink stroke data; analyzing the pixel segmentation map to generate stroke-level semantic information using a pixel-to-stroke conversion model; and processing the digital ink stroke data based on the stroke-level semantic information.

Semantic segmentation for stroke classification in inking application

A sign language translation system may capture infrared images of the formation of a sign language sign or sequence of signs. The captured infrared images may be used to produce skeletal joints data that includes a temporal sequence of 3D coordinates of skeletal joints of hands and forearms that produced the sign language sign(s). A hierarchical bidirectional recurrent neural network may be used to translate the skeletal joints data into a word or sentence of a spoken language. End-to-end sentence translation may be performed using a probabilistic connectionist temporal classification based approach that may not require pre-segmentation of the sequence of signs or post-processing of the translated sentence.

System and apparatus for non-intrusive word and sentence level sign language translation

Existing indoor air quality monitoring technologies focus on measuring which turn out to be effective in increasing people's awareness of air quality. However, the lack of identification of pollution sources is prone to lead to general and monotonous suggestions. In this disclosure, an indoor air quality analytics system is presented that is able to detect pollution events and identify pollution sources in real-time. The system can also forecast personal exposure to air pollution and provide actionable suggestions to help people improve indoor air quality.

Intelligent sensing system for indoor air quality analytics

Described herein is a system and method regarding an interactive drawing tutorial session for creating a drawing using a computer application. A digital image is processed to determine linework and shading information for a plurality of shading levels. A graphical user interface for an interactive drawing tutorial session for creating the drawing by displaying the determined linework to a user. User linework input is received for the drawing based upon the displayed linework information. Feedback is provided to the user regarding the received user linework input (e.g., regarding tilt and/or pressure of electronic stylus). For each of the plurality of shading levels, the determined shading information is displayed to the user. User shading input is received for the drawing based upon the displayed shading information. Feedback is provided to the user regarding the received user shading input (e.g., regarding tilt and/or pressure of electronic stylus).

Growing community of inventors

Lansing, MI, United States of America

Biyi Fang