Amazon Technologies, Inc.

Mmodal IP LLC

Multimodal Technologies, LLC

Lambert Leo Mathias

Imre Attila Kiss

Ying Shi

Frederic Johan Georges Deramat

Ryan Paul Thomas

Juergen Fritsch

Girija Yegnanarayanan

Ruhi Sarikaya

Ariya Rastrow

Ankur Gandhe

Markus Dreyer

Björn Hoffmeister

Ryan Scott Aldrich

Diamond Bishop

Chetan Nagaraj Naik

Bala Murali Krishna Ummaneni

Faisal Ladhak

Arthur Richard Toth

Jesper Mikael Johansson

Giuseppe Di Fabbrizio

Arindam Mandal

Jeffrey Penrod Adams

Chenlei Guo

Stephen Frederick Potter

Xing Fan

Matthew Liang Chaboud

Angeliki Metallinou

Shishir Sridhar Bharathi

Thomas Kollar

Samuel Robert Cavenagh

Weam Abu Zaki

Zheng Chen

Aaron Nicholas Olson

Yuan Ling

A system for processing user utterances and/or text based queries that tracks entities and other context data of a current dialog between the system and the user and can fill slots for new intents of the dialog by performing statistical processing on previously mentioned entities with respect to current slots to be filled. The system may compare a previously mentioned entity to a current slot to be filled using vector representations, such as word embeddings, of the current utterance, dialog history, current intent, name of an entity under consideration, category of the current slot to be filled, distance between the current dialog turn and the dialog turn that mentioned the entity, and other considerations. The individual vectors may be weighted according to an attention operation and processed by a trained decoder to output a score indicating whether the entity in consideration is relevant to the particular slot. In this manner, slots may be filled using entities from previous dialog turns, thus performing statistical anaphora resolution and leading to improved system performance.

Speech processing dialog management

A system is provided for reducing friction during user interactions with a natural language processing system, such as voice assistant systems. The system determines a pre-trained model using dialog session data corresponding to multiple user profiles. The system determines a fine-tuned model using the pre-trained model and a fine-tuning dataset that corresponds to a particular task, such as query rewriting. The system uses the fine-tuned model to process a user input and determine an alternative representation of the input that can result in a desired response from the natural language processing system.

Natural language understanding

In speech processing systems personalization is added in the Natural Language Understanding (NLU) processor by incorporating external knowledge sources of user information to improve entity recognition performance of the speech processing system. Personalization in the NLU is effected by incorporating one or more dictionaries of entries, or gazetteers, with information personal to a respective user, that provide the user's information to permit disambiguation of semantic interpretation for input utterances to improve quality of speech processing results.

Dynamic gazetteers for personalized entity recognition

Features are disclosed for processing a user utterance with respect to multiple subject matters or domains, and for selecting a likely result from a particular domain with which to respond to the utterance or otherwise take action. A user utterance may be transcribed by an automatic speech recognition ('ASR') module, and the results may be provided to a multi-domain natural language understanding ('NLU') engine. The multi-domain NLU engine may process the transcription(s) in multiple individual domains rather than in a single domain. In some cases, the transcription(s) may be processed in multiple individual domains in parallel or substantially simultaneously. In addition, hints may be generated based on previous user interactions and other data. The ASR module, multi-domain NLU engine, and other components of a spoken language processing system may use the hints to more efficiently process input or more accurately generate output.

Architecture for multi-domain natural language processing

Features are disclosed for processing and interpreting natural language, such as interpretations of user utterances, in multi-turn dialog interactions. Context information regarding interpretations of user utterances and system responses to the user utterances can be maintained. Subsequent user utterances can be interpreted using the context information, rather than being interpreted without context. In some cases, interpretations of subsequent user utterances can be merged with interpretations of prior user utterances using a rule-based framework. Rules may be defined to determine which interpretations may be merged and under what circumstances they may be merged.

Context interpretation in natural language processing using previous dialog acts

A system capable of performing natural language understanding (NLU) without the concept of a domain that influences NLU results. The present system uses a hierarchical organizations of intents/commands and entity types, and trained models associated with those hierarchies, so that commands and entity types may be determined for incoming text queries without necessarily determining a domain for the incoming text. The system thus operates in a domain agnostic manner, in a departure from multi-domain architecture NLU processing where a system determines NLU results for multiple domains simultaneously and then ranks them to determine which to select as the result.

Fine-grained natural language understanding

Neural networks may be used in certain automatic speech recognition systems. To improve performance at these neural networks, the present system converts the lattice into a matrix form, thus maintaining certain information included in the lattice that might otherwise be lost while also placing the lattice in a form that may be manipulated by other components to perform operations such as checking ASR results. The matrix representation of the lattice may be transformed into a vector representation by calculations performed at a recurrent neural network (RNN). By representing the lattice as a vector representation the system may perform additional operations, such as ASR results confirmation.

Lattice decoding and result confirmation using recurrent neural networks

An automatic speech recognition (ASR) system may convert an ASR output lattice into a matrix form, thus maintaining certain information included in the lattice that might otherwise be lost in an N-best list output. The matrix representation of the lattice may be encoded using a recurrent neural network (RNN) to create a vector representation of the lattice. The vector representation may then be used by the system to perform additional operations, such as ASR results confirmation.

Lattice encoding using recurrent neural networks

A speech-based system may be configured to receive and act upon spoken requests from a user. In some cases, a spoken request may ask the system to play content such as music without specifying from which of multiple available content sources the music is to be obtained. In response to such a request, the system analyzes feature scores for each of the content sources. The feature scores indicate usage characteristics of the different sources by a current user or groups of users. The features scores for a particular source may be averaged or otherwise combined to create a composite score, and the source having the highest composite score is selected as the source of the requested content.

Selecting from multiple content sources

A system is provided for training an acoustic model for use in speech recognition. In particular, such a system may be used to perform training based on a spoken audio stream and a non-literal transcript of the spoken audio stream. Such a system may identify text in the non-literal transcript which represents concepts having multiple spoken forms. The system may attempt to identify the actual spoken form in the audio stream which produced the corresponding text in the non-literal transcript, and thereby produce a revised transcript which more accurately represents the spoken audio stream. The revised, and more accurate, transcript may be used to train the acoustic model using discriminative training techniques, thereby producing a better acoustic model than that which would be produced using conventional techniques, which perform training based directly on the original non-literal transcript.

Discriminative training of document transcription system

Growing community of inventors

Seattle, WA, United States of America

Lambert Leo Mathias