Amazon Technologies, Inc.

Ankur Gandhe

Ariya Rastrow

Markus Dreyer

Ashish Vishwanath Shenoy

Björn Hoffmeister

Bjorn Hoffmeister

Arindam Mandal

Lambert Leo Mathias

Denis Sergeyevich Filimonov

Raefer Christopher Gabriel

Lizhen Peng

Roland Maximilian Rolf Maas

Gautam Tiwari

Chandra Prakash Khatri

Anushree Venkatesh

Behnam Hedayatnia

Anirudh Raju

Jason Cline

Alok Upadhyay

Faisal Ladhak

Chun Chen

Ahmad Shikib Mehri

Rohit Prasad

Spyridon Matsoukas

Chenlei Guo

Sravan Babu Bodapati

Xing Fan

Katrin Kirchhoff

John Baker

Aaron A Eakin

Anubhav Mishra

Nikko Ström

Ravi Teja Gadde

Angela Sun

Saket Dingliwal

Shaswat Pratap Shah

Julian Chan

Described herein is a system for rescoring automatic speech recognition hypotheses for conversational devices that have multi-turn dialogs with a user. The system leverages dialog context by incorporating data related to past user utterances and data related to the system generated response corresponding to the past user utterance. Incorporation of this data improves recognition of a particular user utterance within the dialog.

Speech recognition using dialog history

Systems and methods for presence ground truth approximation and utilization are disclosed. For example, a system detects the presence of a predefined subject, such as a person associated with a given user profile, and/or determines that authentication criteria for performing an action in association with the user profile has been satisfied. A period of time to associate data is determined, and data of one or more data types is labeled as being associated with the speaker identification event. That data may be formatted and input into one or more models to train those models to more accurately detect presence and/or determine whether authentication of a user profile should succeed.

Authenticating a user profile with devices

Domain-specific parameters may be used for tuning speech processing. A pre-trained transformer-based language model may train domain-specific parameters using domain-specific unlabeled text data. This domain-specific parameters can then be appended to candidate texts produced by a speech model on received speech data and input to the transformer-based language model to score the candidate texts. The scores of the candidate texts determined using the pre-trained transformer-based language model can then be used to select a candidate text for further speech processing.

Domain-specific parameter pre-fixes for tuning automatic speech recognition

Exemplary embodiments relate to adapting a generic language model during runtime using domain-specific language model data. The system performs an audio frame-level analysis, to determine if the utterance corresponds to a particular domain and whether the ASR hypothesis needs to be rescored. The system processes, using a trained classifier, the ASR hypothesis (a partial hypothesis) generated for the audio data processed so far. The system determines whether to rescore the hypothesis after every few audio frames (representing a word in the utterance) are processed by the speech recognition system.

Language model adaptation

Systems and methods described herein relate to adapting a language model for automatic speech recognition (ASR) for a new set of words. Instead of retraining the ASR models, language models and grammar models, the system only modifies one grammar model and ensures its compatibility with the existing models in the ASR system.

Language and grammar model adaptation using model weight data

Presence data determination and utilization

Described herein is a system for prompting a user for clarification when an automatic speech recognition (ASR) system encounters ambiguity with respect to the user's input. The feedback provided by the user is used to retrain machine-learning models and/or to generate new machine-learning models. Based on the type of ambiguity, the system may determine to retrain one or more ASR models that are widely used by the system or to generate/update one or more user-specific models that are used to process inputs from one or more particular users.

Disambiguation in automatic speech processing

Language and grammar model adaptation

A system to perform automatic speech recognition (ASR) using a dynamic language model. Portions of the language model can include a group of probabilities rather than a single probability. At runtime individual probabilities of the group are weighted and combined to create an adjusted probability for the portion of the language model. The adjusted probability can be used for ASR processing. The weights can be determined based on a characteristic of the utterance, for example an associated speechlet/application, the specific user speaking, or other characteristic. By applying the weights at runtime the system can use a single language model to dynamically adjust to different utterance conditions.

Creation of language models for speech recognition

Neural networks may be used in certain automatic speech recognition systems. To improve performance at these neural networks, the present system converts the lattice into a matrix form, thus maintaining certain information included in the lattice that might otherwise be lost while also placing the lattice in a form that may be manipulated by other components to perform operations such as checking ASR results. The matrix representation of the lattice may be transformed into a vector representation by calculations performed at a recurrent neural network (RNN). By representing the lattice as a vector representation the system may perform additional operations, such as ASR results confirmation.

Lattice decoding and result confirmation using recurrent neural networks

An automatic speech recognition (ASR) system may convert an ASR output lattice into a matrix form, thus maintaining certain information included in the lattice that might otherwise be lost in an N-best list output. The matrix representation of the lattice may be encoded using a recurrent neural network (RNN) to create a vector representation of the lattice. The vector representation may then be used by the system to perform additional operations, such as ASR results confirmation.

Lattice encoding using recurrent neural networks

A language model for automatic speech processing, such as a finite state transducer (FST) may be configured to incorporate information about how a particular word sequence (N-gram) may be used in a similar manner from another N-gram. A score of a component of the FST (such as an arc or state) relating to the first N-gram may be based on information of the second N-gram. Further, the FST may be configured to have an arc between a state of the first N-gram and a state of the second N-gram to allow for cross N-gram back off, rather than backoff from a larger N-gram to a smaller N-gram during traversal of the FST during speech processing.

Automatic speech recognition incorporating word usage information

An automatic speech recognition ('ASR') system produces, for particular users, customized speech recognition results by using data regarding prior interactions of the users with the system. A portion of the ASR system (e.g., a neural-network-based language model) can be trained to produce an encoded representation of a user's interactions with the system based on, e.g., transcriptions of prior utterances made by the user. This user-specific encoded representation of interaction history is then used by the language model to customize ASR processing for the user.

Speech processing with learned representation of user interaction history

Growing community of inventors

Bothell, WA, United States of America

Ankur Gandhe