Massachusetts Institute of Technology

Nippon Telegraph and Telephone Corporation

James Robert Glass

David F Harwath

Kunio Kashino

Anantha P Chandrakasan

Takahito Kawanishi

Akisato Kimura

Michael R Price

Yasunori Ohishi

William D Goldenthal

Mandy Barrett Korpusik

Takaaki Hori

I Lee Hetherington

Timothy J Hazen

Jon Rong-Wei Yi

Irvine Lee Hetherington

ILee Hetherington

A decoder includes a feature extraction circuit for calculating one or more feature vectors. An acoustic model circuit is coupled to receive one or more feature vectors from and assign one or more likelihood values to the one or more feature vectors. A memory architecture that utilizes on-chip state lattices and an off-chip memory for storing states of transition of the decoder is used to reduce reading and writing to the off-chip memory. The on-chip state lattice is populated with at least one of the states of transition stored in the off-chip memory. An on-chip word is generated from a snapshot from the on-chip state lattice. The on-chip state lattice and the on-chip word lattice act as an on-chip cache to reduce reading and writing to the off-chip memory.

Low-power automatic speech recognition device

A learning device calculates a feature of each data included in a pair of datasets in which two modalities among a plurality of modalities are combined, using a model that receives data on a corresponding modality among the modalities and outputs a feature obtained by mapping the received data into an embedding space. The learning device then selects similar data similar to each target data that is data on a first modality in a first dataset of the datasets, from data on a second modality included in a second dataset of the datasets. The learning device further updates a parameter of the model such that the features of the data in the pair included in the first and the second datasets are similar to one another, and the feature of data paired with the target data is similar to the feature of data paired with the similar data.

Learning device, learning method, and learning program for images and sound which uses a similarity matrix

A learning device calculates an image feature using a model (image encoder) that receives an image and outputs the image feature obtained by mapping the image into a latent space. The learning device calculates an audio feature using a model (audio encoder) that receives a speech in a predetermined language and outputs the audio feature obtained by mapping the speech into the latent space, and that includes a neural network provided with a self-attention mechanism. The learning device updates parameters of the models used by an image feature calculation unit and an audio feature calculation unit such that the image feature of a first image is similar to the audio feature of a speech corresponding to the first image.

Learning device, learning method, learning program, retrieval device, retrieval method, and retrieval program

A decoder comprises a feature extraction circuit for calculating one or more feature vectors; an acoustic model circuit coupled to receive one or more feature vectors from said feature extraction circuit and assign one or more likelihood values to the one or more feature vectors; a memory for storing states of transition of the decoder; and a search circuit for receiving an input from said acoustic model circuit corresponding to the one or more likelihood values based upon the one or more feature vectors, and for choosing states of transition from the memory based on the input from said acoustic model.

A system for associating a string of natural language with items in a relational database includes a first subsystem having a pre-trained first artificial neural network configured to apply a semantic tag selected from a predefined set of semantic labels to a segment of a plurality of tokens representing the string of natural language. A second subsystem includes a second artificial neural network configured to convert the plurality of labeled tokens into a first multi-dimensional vector representing the string of natural language. A third subsystem is configured to rank the first multi-dimensional vector against a second multi-dimensional vector representing a plurality of items in the relational database.

System and method for semantic mapping of natural language input to database entries via convolutional neural networks

An approach to joint acoustic and visual processing associates images with corresponding audio signals, for example, for the retrievals of images according to voice queries. A set of paired images and audio signals are processed without requiring transcription, segmentation, or annotation of either the images or the audio. This processing of the paired images and audio is used to determine parameters of an image processor and an audio processor, with the outputs of these processors being comparable to determine a similarity across acoustic and visual modalities. In some implementations, the image processor and the audio processor make use of deep neural networks. Further embodiments associate parts of images with corresponding parts of audio signals.

Joint acoustic and visual processing

A speech data retrieval apparatus () includes a speech database (), a speech recognition unit (), a confusion network creation unit (), an inverted index table creation unit (), a query input unit (), a query conversion unit () and a label string check unit (). The speech recognition unit () reads speech data from the speech database (), carries out a speech recognition process with respect to the read speech data, and outputs a result of speech recognition process as a lattice in which a phoneme, a syllable, or a word is a base unit. The confusion network creation unit () creates a confusion network based on the output lattice and outputs the result of speech recognition process as the confusion network. The inverted index table creation unit () creates an inverted index table based on the output confusion network. The query input unit () receives a query input by a user, carries out a speech recognition process with respect to the received query, and outputs a result of speech recognition process as a character string. The query conversion unit () converts the output character string into a label string in which a phoneme, a syllable, or a word is a base unit. The label string check unit () checks the label string against the inverted index table and retrieves speech data which is included in both of the label string and the speech database ().

Speech data retrieval apparatus, speech data retrieval method, speech data retrieval program and computer usable medium having computer readable speech data retrieval program embodied therein

A method for concatenative speech synthesis includes a processing stage that selects segments based on their symbolic labeling in an efficient graph-based search, which uses a finite-state transducer formalism. This graph-based search uses a representation of concatenation constraints and costs that does not necessarily grow with the size of the source corpus thereby limiting the increase in computation required for the search as the size of the source corpus increases. In one application of this method, multiple alternative segment sequences are generated and a best segment sequence is then be selected using characteristics that depend on specific signal characteristics of the segments.

Concatenative speech synthesis using a finite-state transducer

A feature-based speech recognizer having a probabilistic linguistic processor provides word matching based on the entire space of feature vectors. In this manner, the errors and inaccuracies associated with the heretofore known feature-based speech recognizers, which provided word matching on less than the entire space of feature vectors, are overcome, thereby resulting in improved-accuracy speech recognition. The word matching may be on feature vectors computed either from segments or from landmarks or from both segments and landmarks. For word matching on segment-based feature vectors, acoustic likelihoods may be normalized by extra-acoustic likelihoods defined by at least one extra-acoustic ('not' or 'anti') model. Context-dependent and context-independent acoustic models may be employed.

Feature-based speech recognizer having probabilistic linguistic

Phonetic recognition is provided by capturing dynamical behavior and statistical dependencies of the acoustic attributes used to represent a subject speech waveform. A segment based framework is employed. Temporal behavior is modelled explicitly by creating dynamic templates, called tracks, of the acoustic attributes used to represent the speech waveform, and by generating the estimation of the acoustic spatio-temporal correlation structure. An error model represents this estimation as the temporal and spatial correlations between the input speech waveform and track generated speech segment. Models incorporating these two components (track and error estimation) are created for both phonetic units and for phonetic transitions. Phonetic contextual influences are accounted for by merging context-dependent tracks and pooling error statistics over the different contexts. This allows for a large number of contextual models without compromising the robustness of the statistical parameter estimates. The transition models also supply contextual information.

Growing community of inventors

Cambridge, MA, United States of America

James Robert Glass