Amazon Technologies, Inc.

Rawles LLC

Bjorn Hoffmeister

Ariya Rastrow

Shiv Naga Prasad Vitaladevuni

Nikko Strom

Rohit Prasad

Spyridon Matsoukas

Roland Maximilian Rolf Maas

Sri Venkata Surya Siva Rama Krishna Garimella

Arindam Mandal

Ryan Paul Thomas

Minhua Wu

Hugh Evan Secker-Walker

Kenneth John Basye

Shiva Kumar Sundaram

Eli Joshua Fidler

Jeffrey C O'Neill

Rajiv Ramachandran

Sankaran Panchapagesan

Kenichi Kumatani

Baiyang Liu

Stan Weidner Salvador

Ming Sun

Ankur Gandhe

Aaron A Eakin

Brian John King

Diamond Bishop

Roland Maas

Sree Hari Krishnan Parthasarathi

Sanjeev Mishra

James G Droppo

Robert Ayrapetian

Jeffrey Penrod Adams

Scott Ian Blanksteen

Xing Fan

Yuzo Watanabe

Andrew Smith

Aparna Khare

Karthik Ramakrishnan

Gautam Tiwari

Grant Strimel

Michael Reese Bastian

Nikko Ström

Paul Joseph Schaffert

Veerdhawal Pande

Yuzong Liu

Michael Maximilian Emanuel Bisani

Spyros Matsoukas

I-Fan Chen

Sri Harish Reddy Mallidi

Yiming Wang

Tongfei Chen

Maarten Van Segbroeck

Jeffrey Cornelius O'NEILL

Techniques for reducing latency in processing of audio data, where the latency may be caused in detecting audio of interest in the audio data, are described. A device that captures audio data may include a detection component to determine when the audio data includes audio of interest (e.g., device-directed speech), and an audio embedding generator to generate embedding vectors for the captured audio data while the detection component processes the audio data. The device may generate an embedding vector for audio data captured at the device for a duration of time; determine, at the end of the duration of time, that the audio data represents audio of interest; and send the embedding vector to an audio processing component (e.g., an automatic speech recognition component) for processing.

Low latency audio processing techniques

A speech interface device is configured to detect an interrupt event and process a voice command without detecting a wakeword. The device includes on-device interrupt architecture configured to detect when device-directed speech is present and send audio data to a remote system for speech processing. This architecture includes an interrupt detector that detects an interrupt event (e.g., device-directed speech) with low latency, enabling the device to quickly lower a volume of output audio and/or perform other actions in response to a potential voice command. In addition, the architecture includes a device directed classifier that processes an entire utterance and corresponding semantic information and detects device-directed speech with high accuracy. Using the device directed classifier, the device may reject the interrupt event and increase a volume of the output audio or may accept the interrupt event, causing the output audio to end and performing speech processing on the audio data.

Device-directed utterance detection

A speech-processing system may determine potential endpoints in a user's speech. Such endpoint prediction may include determining a potential endpoint in a stream of audio data, and may additionally including determining an endpoint score representing a likelihood that the potential endpoint represents an end of speech representing a complete user input. When the potential endpoint has been determined, the system may publish a transcript of speech that preceded the potential endpoint, and send it to downstream components. The system may continue to transcribe audio data and determine additional potential endpoints while the downstream components process the transcript. The downstream components may determine whether the transcript is complete; e.g., represents the entirety of the user input. Final endpoint determinations may be made based on the results of the downstream processing including automatic speech recognition, natural language understanding, etc.

Endpointing in speech processing

Exemplary embodiments relate to adapting a generic language model during runtime using domain-specific language model data. The system performs an audio frame-level analysis, to determine if the utterance corresponds to a particular domain and whether the ASR hypothesis needs to be rescored. The system processes, using a trained classifier, the ASR hypothesis (a partial hypothesis) generated for the audio data processed so far. The system determines whether to rescore the hypothesis after every few audio frames (representing a word in the utterance) are processed by the speech recognition system.

Language model adaptation

Features are disclosed for detecting words in audio using contextual information in addition to automatic speech recognition results. A detection model can be generated and used to determine whether a particular word, such as a keyword or 'wake word,' has been uttered. The detection model can operate on features derived from an audio signal, contextual information associated with generation of the audio signal, and the like. In some embodiments, the detection model can be customized for particular users or groups of users based usage patterns associated with the users.

Wake word detection modeling

Techniques for speech processing using a deep neural network (DNN) based acoustic model front-end are described. A new modeling approach directly models multi-channel audio data received from a microphone array using a first model (e.g., multi-geometry/multi-channel DNN) that is trained using a plurality of microphone array geometries. Thus, the first model may receive a variable number of microphone channels, generate multiple outputs using multiple microphone array geometries, and select the best output as a first feature vector that may be used similarly to beamformed features generated by an acoustic beamformer. A second model (e.g., feature extraction DNN) processes the first feature vector and transforms it to a second feature vector having a lower dimensional representation. A third model (e.g., classification DNN) processes the second feature vector to perform acoustic unit classification and generate text data. The DNN front-end enables improved performance despite a reduction in microphones.

Deep multi-channel acoustic modeling using multiple microphone array geometries

A system configured to process speech commands may classify incoming audio as desired speech, undesired speech, or non-speech. Desired speech is speech that is from a same speaker as reference speech. The reference speech may be obtained from a configuration session or from a first portion of input speech that includes a wakeword. The reference speech may be encoded using a recurrent neural network (RNN) encoder to create a reference feature vector. The reference feature vector and incoming audio data may be processed by a trained neural network classifier to label the incoming audio data (for example, frame-by-frame) as to whether each frame is spoken by the same speaker as the reference speech. The labels may be passed to an automatic speech recognition (ASR) component which may allow the ASR component to focus its processing on the desired speech.

Anchored speech detection and speech recognition

Techniques for speech processing using a deep neural network (DNN) based acoustic model front-end are described. A new modeling approach directly models multi-channel audio data received from a microphone array using a first model (e.g., multi-channel DNN) that takes in raw signals and produces a first feature vector that may be used similarly to beamformed features generated by an acoustic beamformer. A second model (e.g., feature extraction DNN) processes the first feature vector and transforms it to a second feature vector having a lower dimensional representation. A third model (e.g., classification DNN) processes the second feature vector to perform acoustic unit classification and generate text data. These three models may be jointly optimized for speech processing (as opposed to individually optimized for signal enhancement), enabling improved performance despite a reduction in microphones and a reduction in bandwidth consumption during real-time processing.

Deep multi-channel acoustic modeling

A speech-processing system capable of receiving and processing audio data to determine if the audio data includes speech that was intended for the system. Non-system directed speech may be filtered out while system-directed speech may be selected for further processing. A system-directed speech detector may use a trained machine learning model (such as a deep neural network or the like) to process a feature vector representing a variety of characteristics of the incoming audio data, including the results of automatic speech recognition and/or other data. Using the feature vector the model may output an indicator as to whether the speech is system-directed. The system may also incorporate other filters such as voice activity detection prior to speech recognition, or the like.

Detecting system-directed speech

An approach to wakeword detection uses an explicit representation of non-wakeword speech in the form of subword (e.g., phonetic monophone) units that do not necessarily occur in the wakeword and that broadly represent general speech. These subword units are arranged in a 'background' model, which at runtime essentially competes with the wakeword model such that a wakeword is less likely to be declare as occurring when the input matches that background model well. An HMM may be used with the model to locate possible occurrences of the wakeword. Features are determined from portions of the input corresponding to subword units of the wakeword detected using the HMM. A secondary classifier is then used to process the features to yield a decision of whether the wakeword occurred.

Monophone-based background modeling for wakeword detection

A system configured to recognize text represented by speech may determine that a first portion of audio data corresponds to speech from a first speaker and that a second portion of audio data corresponds to speech from the first speaker and a second speaker. Features of the first portion are compared to features of the second portion to determine a similarity therebetween. Based on this similarity, speech from the first speaker is distinguished from speech from the second speaker and text corresponding to speech from the first speaker is determined.

Speech detection and speech recognition

Keyword detection modeling using contextual information

Growing community of inventors

Seattle, WA, United States of America

Bjorn Hoffmeister