Amazon Technologies, Inc.

Ariya Rastrow

Ankur Gandhe

Bjorn Hoffmeister

Rohit Prasad

Nikko Strom

Denis Sergeyevich Filimonov

Gautam Tiwari

Roland Maximilian Rolf Maas

Arindam Mandal

Eli Joshua Fidler

Aaron A Eakin

Sri Venkata Surya Siva Rama Krishna Garimella

Shaun Nidhiri Joseph

Spyridon Matsoukas

Hugh Evan Secker-Walker

Markus Dreyer

Grant Strimel

Björn Hoffmeister

Aaron Lee Mathers Challenner

Nikko Ström

Andrew Oberlin

James G Droppo

Baiyang Liu

Tony Roy Hardie

William Folwell Barton

Lambert Leo Mathias

Stephen Frederick Potter

Yuzo Watanabe

Madan Mohan Rao Jampani

Angeliki Metallinou

Siddhartha Reddy Jonnalagadda

Arushan Rajasekaram

Prakash Krishnan

Raefer Christopher Gabriel

Jonathan Jenner Macoskey

Chandra Prakash Khatri

Ashish Vishwanath Shenoy

Anirudh Raju

Diamond Bishop

Anushree Venkatesh

Behnam Hedayatnia

Veerdhawal Pande

Vincent Auvray

Faisal Ladhak

Sanjeev Mishra

Ahmad Shikib Mehri

Minmin Shen

Chun Chen

Shiv Naga Prasad Vitaladevuni

Jeffrey Penrod Adams

Chenlei Guo

Xing Fan

Stanislaw Ignacy Pasko

Kevin Crews

Frederick Victor Weber

Alexander David Rosen

Andrew Smith

Sankaran Panchapagesan

Ying Shi

Aaron Challenner

Jeffrey Paul Lilly

Michael Reese Bastian

Venkata Kishore Nandury

Roland Maas

Michal Kowalczuk

Thomas Edward Taylor

David Chi-Wai Tang

Ravi Teja Gadde

Angela Sun

Spyros Matsoukas

Pankaj Sitpure

Alok Ulhas Parlikar

Nicholas Matthew Jutila

Samarth Batra

Prasanna H Sridhar

Paul Anthony Bernhardt

Cagdas Bak

Pawel Zelazko

Josey Diego Sandoval

Nishtha Gupta

Bonan Zheng

Amotz Maimon

Shaswat Pratap Shah

Maarten Van Segbroeck

Gurpreet Singh Chadha

Milind Mukesh Rao

Santosh Kumar Patro

Buddha Puneeth Nandanoor

Anand Mohan

Gopinath Chennupati

Prahalad Venkataramanan

Julian Chan

Sonal Pareek

Described herein is a system for rescoring automatic speech recognition hypotheses for conversational devices that have multi-turn dialogs with a user. The system leverages dialog context by incorporating data related to past user utterances and data related to the system generated response corresponding to the past user utterance. Incorporation of this data improves recognition of a particular user utterance within the dialog.

Speech recognition using dialog history

A first neural network (NN) model may generate labels for training a second NN model. The second NN model may represent instances of a NN model operating on multiple different devices (e.g., decentralized user and/or edge devices). The system may include using a 'teacher' model to process data received by one or more of the devices to generate a labeled dataset. The system may use the labeled dataset and a “student” model to calculate gradient data for updating the student model. The student model may be the same or similar to NN model instances operating on the devices. The system may validate the updated student model to determine, for example, whether it exhibits improved performance when processing the newly received data and/or historical data. The system may distribute the validated update to the devices.

Continuous learning for machine learning models

Techniques for reducing latency in processing of audio data, where the latency may be caused in detecting audio of interest in the audio data, are described. A device that captures audio data may include a detection component to determine when the audio data includes audio of interest (e.g., device-directed speech), and an audio embedding generator to generate embedding vectors for the captured audio data while the detection component processes the audio data. The device may generate an embedding vector for audio data captured at the device for a duration of time; determine, at the end of the duration of time, that the audio data represents audio of interest; and send the embedding vector to an audio processing component (e.g., an automatic speech recognition component) for processing.

Low latency audio processing techniques

A speech interface device is configured to detect an interrupt event and process a voice command without detecting a wakeword. The device includes on-device interrupt architecture configured to detect when device-directed speech is present and send audio data to a remote system for speech processing. This architecture includes an interrupt detector that detects an interrupt event (e.g., device-directed speech) with low latency, enabling the device to quickly lower a volume of output audio and/or perform other actions in response to a potential voice command. In addition, the architecture includes a device directed classifier that processes an entire utterance and corresponding semantic information and detects device-directed speech with high accuracy. Using the device directed classifier, the device may reject the interrupt event and increase a volume of the output audio or may accept the interrupt event, causing the output audio to end and performing speech processing on the audio data.

Device-directed utterance detection

A speech-processing system may determine potential endpoints in a user's speech. Such endpoint prediction may include determining a potential endpoint in a stream of audio data, and may additionally including determining an endpoint score representing a likelihood that the potential endpoint represents an end of speech representing a complete user input. When the potential endpoint has been determined, the system may publish a transcript of speech that preceded the potential endpoint, and send it to downstream components. The system may continue to transcribe audio data and determine additional potential endpoints while the downstream components process the transcript. The downstream components may determine whether the transcript is complete; e.g., represents the entirety of the user input. Final endpoint determinations may be made based on the results of the downstream processing including automatic speech recognition, natural language understanding, etc.

Endpointing in speech processing

Techniques for using multiple machine learning (ML) models, with varying compute costs, for ASR processing is described. The system may include an arbitrator component configured to determine which ML model is to be used to process an audio frame from a sequence of audio frames representing a spoken natural language input. The arbitrator component may switch between the ML models, on a frame-by-frame basis, to reduce an overall compute cost for the entire spoken natural language input. The outputs of the different ML models may be combined to determine the final output for the entire spoken natural language input.

Speech processing techniques

A natural language system may be configured to act as a participant in a conversation between two users. The system may determine when a user expression such as speech, a gesture, or the like is directed from one user to the other. The system may processing input data related the expression (such as audio data, input data, language processing result data, conversation context data, etc.) to determine if the system should interject a response to the user-to-user expression. If so, the system may process the input data to determine a response and output it. The system may track that response as part of the data related to the ongoing conversation.

Dialog management for multiple users

Exemplary embodiments relate to adapting a generic language model during runtime using domain-specific language model data. The system performs an audio frame-level analysis, to determine if the utterance corresponds to a particular domain and whether the ASR hypothesis needs to be rescored. The system processes, using a trained classifier, the ASR hypothesis (a partial hypothesis) generated for the audio data processed so far. The system determines whether to rescore the hypothesis after every few audio frames (representing a word in the utterance) are processed by the speech recognition system.

Language model adaptation

A system that is capable of resolving anaphora using timing data received by a local device. A local device outputs audio representing a list of entries. The audio may represent synthesized speech of the list of entries. A user can interrupt the device to select an entry in the list, such as by saying 'that one.' The local device can determine an offset time representing the time between when audio playback began and when the user interrupted. The local device sends the offset time and audio data representing the utterance to a speech processing system which can then use the offset time and stored data to identify which entry on the list was most recently output by the local device when the user interrupted. The system can then resolve anaphora to match that entry and can perform additional processing based on the referred to item.

Some devices may perform processing using machine learning models trained at a centralized system and distributed to the device. The centralized system may update the machine learning model and distribute the update to the device (or devices). To reduce the size of an update, the centralized system may train a model update object, which may be smaller in size than the model itself and thus more suitable for sending to the device(s). A device may receive the model update object and use it to update the on-device machine learning model; for example, by changing some parameters of the model. Parameters left unchanged during the update may retain their previous value. Thus, using the model update object to update the on-device model may result in a more accurate updated model when compared to sending an updated model compressed to a size similar to that of the model update object.

Updating models with trained model update objects

Some speech processing systems may handle some commands on-device rather than sending the audio data to a second device or system for processing. The first device may have limited speech processing capabilities sufficient for handling common language and/or commands, while the second device (e.g., an edge device and/or a remote system) may call on additional language models, entity libraries, skill components, etc. to perform additional tasks. An intermediate data generator may facilitate dividing speech processing operations between devices by generating a stream of data that includes a first-pass ASR output (e.g., a word or sub-word lattice) and other characteristics of the audio data such as whisper detection, speaker identification, media signatures, etc. The second device can perform the additional processing using the data stream; e.g., without using the audio data. Thus, privacy may be enhanced by processing the audio data locally without sending it to other devices/systems.

Intermediate data for inter-device speech processing

Systems and methods described herein relate to adapting a language model for automatic speech recognition (ASR) for a new set of words. Instead of retraining the ASR models, language models and grammar models, the system only modifies one grammar model and ensures its compatibility with the existing models in the ASR system.

Language and grammar model adaptation using model weight data

This disclosure proposes systems and methods for leveraging entity-related language models in speech processing. A system can receive audio data corresponding to an utterance and perform automatic speech recognition (ASR) on a first portion of the audio data using a general language model. Based on the results, the system may identify a specific language model for processing a second portion of the audio data. The specific language model may include entities belonging to a common subject or class. The specific language model may, in some cases, provide better results than the general language model. While the general language model may describe a whole sentence, the specific language model may describe only a portion of a sentence. Thus, a top-level model may 'activate' the specific language model when it may provide useful results. The resulting data may include results from both the general language model and the specific language model.

Entity language models for speech processing

A speech interface device is configured to receive response data from a remote speech processing system for responding to user speech. This response data may be enhanced with information such as remote NLU data. The response data from the remote speech processing system may be compared to local NLU data to improve a speech processing model on the device. Thus, the device may perform supervised on-device learning based on the remote NLU data. The device may determine differences between the updated speech processing model and an original speech processing model received from the remote system and may send data indicating these differences to the remote system. The remote system may aggregate data received from a plurality of devices and may generate an improved speech processing model.

On-device learning in a hybrid speech processing system

Techniques for an interactive turn-based reading experience are described. A system may take turns reading content, such as a book, with a user. The system may process audio data representing a user reading a portion of the content, determine reading evaluation data, and determine how to proceed for the next turn based on the reading evaluation data. For example, based on the reading evaluation data, the system may read a portion of the content by outputting synthesized speech representing the content, may ask the user re-read a portion of the content, or may ask the user to read a different, smaller portion of the content.

Growing community of inventors

Seattle, WA, United States of America

Ariya Rastrow