Michigan State University

Mi Zhang

Xiao Zeng

Biyi Fang

Kai Cao

Haochen Sun

Taiwoo Park

The present disclosure describes systems and methods for implementing input-adaptive neural networks. Several approaches and systems are presented for creating flexible and computationally-efficient deep neural networks that adapt their complexity based upon input data such as video streams. The approach may use fine-grained design and an automatic approach for generating optimal network architectures based on early-exit mechanisms. A neural network modified according to such principles can be implemented in various devices including mobile vision systems such as smartphones, drones, and head-mounted augmented reality devices.

Systems and methods for implementing flexible, input-adaptive deep learning neural networks

Systems and methods are disclosed which allow mobile devices, and other resource constrained applications, to more efficiently and effectively utilize deep learning neural networks using only (or primarily) local resources. These systems and methods take the dynamics of runtime resources into account to enable resource-aware, multi-tenant on-device deep learning for artificial intelligence functions for use in tasks like mobile vision systems. The multi-capacity framework enables deep learning models to offer flexible resource-accuracy trade-offs and other similar balancing of performance and resources consumed. At runtime, various systems disclosed herein may dynamically select the optimal resource-accuracy trade-off for each deep learning model to fit the model's resource demand to the system's available runtime resources and the needs of the task being performed by the model. In doing so, systems and methods disclosed herein can efficiently utilize the limited resources in mobile systems to maximize performance of multiple concurrently running neural network-based applications.

Systems and methods for providing flexible, multi-capacity models for use of deep neural networks in mobile devices

A system may perform speech enhancement of audio data in real-time by suppressing noise components that are present in the audio data while preserving speech components. The system may include an in-ear module and a separate signal processing module that is wirelessly communicatively coupled to the in-ear module. The system may include non-negative matrix factorization (NMF) dictionaries capable of identifying frequency band components associated with speech and frequency band components associated with noise. The NMF dictionaries may be trained using voice samples and noise samples. The NMF dictionaries may be applied to noisy speech data to produce an NMF representation of the speech data which may then be applied using a dynamic mask to the noisy speech data in order to suppress the noise components of the noisy speech data and produce speech enhanced data.

System and apparatus for real-time speech enhancement in noisy environments

A sign language translation system may capture infrared images of the formation of a sign language sign or sequence of signs. The captured infrared images may be used to produce skeletal joints data that includes a temporal sequence of 3D coordinates of skeletal joints of hands and forearms that produced the sign language sign(s). A hierarchical bidirectional recurrent neural network may be used to translate the skeletal joints data into a word or sentence of a spoken language. End-to-end sentence translation may be performed using a probabilistic connectionist temporal classification based approach that may not require pre-segmentation of the sequence of signs or post-processing of the translated sentence.

System and apparatus for non-intrusive word and sentence level sign language translation

Existing indoor air quality monitoring technologies focus on measuring which turn out to be effective in increasing people's awareness of air quality. However, the lack of identification of pollution sources is prone to lead to general and monotonous suggestions. In this disclosure, an indoor air quality analytics system is presented that is able to detect pollution events and identify pollution sources in real-time. The system can also forecast personal exposure to air pollution and provide actionable suggestions to help people improve indoor air quality.

Intelligent sensing system for indoor air quality analytics

A system and method is provided that utilizes deep learning, including convolutional neural networks, to identify subject objects in unconstrained user images such as unknown pills. An image of, e.g., a pill, may be captured and subsequently processed using deep learning models to identify the pill. The deep learning models may be optimized to have a small footprint (in terms of computational and memory resources) suitable for a resource-limited device such as a smartphone while retaining a high object recognition accuracy. Each such model may also be run on modified versions of the unconstrained image, for example on color, greyscale, and gradient images, to focus the models on different distinguishing features of the object.

Growing community of inventors

Okemos, MI, United States of America

Mi Zhang