Amazon Technologies, Inc.

Spyridon Matsoukas

Arindam Mandal

Rohit Prasad

Shiv Naga Prasad Vitaladevuni

Chao Wang

Aparna Khare

Bjorn Hoffmeister

Ming Sun

Shiva Kumar Sundaram

Nikko Strom

Sundararajan Srinivasan

Ruhi Sarikaya

Rahul Gupta

Ariya Rastrow

Stephen Frederick Potter

Young-Bum Kim

Kenneth John Basye

Rajiv Ramachandran

Frédéric Johan Georges Deramat

Kerry Hammil

Yixin Gao

Kenichi Kumatani

Minhua Wu

Chengwei Su

Krishna Subramanian

Anish N Shah

Sankaranarayanan Ananthakrishnan

Melanie Chie Bomke Gens

Vishwanathan Krishnamoorthy

Anirudh Raju

Avnish Sikka

Sri Venkata Surya Siva Rama Krishna Garimella

Shamitha Somashekar

Chieh-Chi Kao

Thibaud Senechal

Simon Peter Reavely

Rahul Goel

Shirin Saleem

Kellen Gillespie

Thomas Kollar

Cengiz Erbas

Nikko Ström

Kelly James Vanee

Alexandra R Shapiro

Gregory Michael Hart

Chenlei Guo

Xing Fan

Ankur Gandhe

Viktor Rozgic

Denis Sergeyevich Filimonov

Premkumar Natarajan

Markus Dreyer

Sankaran Panchapagesan

Roland Maximilian Rolf Maas

Qingming Tang

Melanie C B Gens

Vasiliy Radostev

Sai Sailesh Kopuri

Casey Stuart Smith

Chungnam Chan

John Walter McDonough, Jr

Mallory McManamon

Kavya Ravikumar

Xiaojiang Huang

Julian Chan

Volker Sebastian Leutnant

Sri Venkata Surya Siva Rama Krishna Garimell

Thibaud Senechai

John Moore

Yi Luo

Mayur Himmatbhai Dabhi

Qingwen Ye

Apurv Verma

Jwala Dhamala

Avni Khatri

Srinivasan Rengarajan Veeravanallur

Seyed Omid Razavi

Jie Bao

Muddu Krishna Chintha

Yugang Li

Priti Bisaria

Luca Soldaini

John Will Crimmins

John Pelak

Sri Harish Reddy Mallidi

Venkata Naga Krishna Chaitanya Puvvada

A system capable of performing natural language understanding (NLU) on utterances including complex command structures such as sequential commands (e.g., multiple commands in a single utterance), conditional commands (e.g., commands that are only executed if a condition is satisfied), and/or repetitive commands (e.g., commands that are executed until a condition is satisfied). Audio data may be processed using automatic speech recognition (ASR) techniques to obtain text. The text may then be processed using machine learning models that are trained to parse text of incoming utterances. The models may identify complex utterance structures and may identify what command portions of an utterance go with what conditional statements. Machine learning models may also identify what data is needed to determine when the conditionals are true so the system may cause the commands to be executed (and stopped) at the appropriate times.

Processing complex utterances for natural language understanding

A system may be configured to receive and process various signals to generate a natural language description of a user's environment, called situational context data. The signals may include sensor data, device status, user activity, user input, and/or inferences made using such data. The situational context data may express a user-centric description of the user's environment; for example: 'User is taking a walk in the park on a sunny afternoon' or “activity: driving location: highway”, etc. The system may send the situational context data to various system components that may, for example, process speech, select applications/skills for handling user inputs, and/or that implement those applications/skills. The applications/skills may use the situational context data to provide recommendations, generate responses, and/or perform actions that are more relevant to the user's current environment.

Determining device context

Systems and methods for utilizing microphone array information for acoustic modeling are disclosed. Audio data may be received from a device having a microphone array configuration. Microphone configuration data may also be received that indicates the configuration of the microphone array. The microphone configuration data may be utilized as an input vector to an acoustic model, along with the audio data, to generate phoneme data. Additionally, the microphone configuration data may be utilized to train and/or generate acoustic models, select an acoustic model to perform speech recognition with, and/or to improve trigger sound detection.

Speech processing optimizations based on microphone array

Devices and techniques are generally described for cohort determination in natural language processing. In various examples, a first natural language input to a natural language processing system may be determined. The first natural language input may be associated with a first account identifier. A first machine learning model may determine first data representing one or more words of the first natural language input. A second machine learning model may determine second data representing one or more acoustic characteristics of the first natural language input. Third data may be determined, the third data including a predicted performance for processing the first natural language input by the natural language processing system. The third data may be determined based on the first data representation and the second data representation.

Cohort determination in natural language processing

An acoustic event detection system may employ self-supervised federated learning to update encoder and/or classifier machine learning models. In an example operation, an encoder may be pre-trained to extract audio feature data from an audio signal. A decoder may be pre-trained to predict a subsequent portion of audio data (e.g., a subsequent frame of audio data represented by log filterbank energies). The encoder and decoder may be trained using self-supervised learning to improve the decoder's predictions and, by extension, the quality of the audio feature data generated by the encoder. The system may apply federated learning to share encoder updates across user devices. The system may fine-tune the classifier to improve inferences based on the improved audio feature data. The system may distribute classifier updates to the user device(s) to update the on-device classifier.

Self-supervised federated learning

Techniques for enrolling a user in a system's user recognition functionality without requiring the user speak particular speech are described. The system may determine characteristics unique to a user input. The system may generate an implicit voice profile from user inputs having similar characteristics. After an implicit voice profile is generated, the system may receive a user input having speech characteristics similar to that of the implicit voice profile. The system may ask the user if the user wants the system to associate the implicit voice profile with a particular user identifier. If the user responds affirmatively, the system may request an identifier of a user profile (e.g., a user name). In response to receiving the user's name, the system may identify a user profile associated with the name and associate the implicit voice profile with the user profile, thereby converting the implicit voice profile into an explicit voice profile.

Speech based user recognition

A system processes audio data to detect when it includes a representation of a wakeword or of an acoustic event. The system may receive or determine acoustic features for the audio data, such as log-filterbank energy (LFBE). The acoustic features may be used by a first, wakeword-detection model to detect the wakeword; the output of this model may be further processed using a softmax function, to smooth it, and to detect spikes. The same acoustic features may be also be used by a second, acoustic-event-detection model to detect the acoustic event; the output of this model may be further processed using a sigmoid function and a classifier. Another model may be used to extract additional features from the LFBE data; these additional features may be used by the other models.

Wakeword and acoustic event detection

A speech-capture device can capture audio data during wakeword monitoring and use the audio data to determine if a user is present nearby the device, even if no wakeword is spoken. Audio such as speech, human originating sounds (e.g., coughing, sneezing), or other human related noises (e.g., footsteps, doors closing) can be used to detect audio. Audio frames are individually scored as to whether a human presence is detected in the particular audio frames. The scores are then smoothed relative to nearby frames to create a decision for a particular frame. Presence information can then be sent according to a periodic schedule to a remote device to create a presence 'heartbeat' that regularly identifies whether a user is detected proximate to a speech-capture device.

User presence detection

Features are disclosed for detecting words in audio using contextual information in addition to automatic speech recognition results. A detection model can be generated and used to determine whether a particular word, such as a keyword or 'wake word,' has been uttered. The detection model can operate on features derived from an audio signal, contextual information associated with generation of the audio signal, and the like. In some embodiments, the detection model can be customized for particular users or groups of users based usage patterns associated with the users.

Wake word detection modeling

A speech-processing system capable of receiving and processing audio data to determine if the audio data includes speech that was intended for the system. Non-system directed speech may be filtered out while system-directed speech may be selected for further processing. A system-directed speech detector may use a trained machine learning model (such as a deep neural network or the like) to process a feature vector representing a variety of characteristics of the incoming audio data, including the results of automatic speech recognition and/or other data. Using the feature vector the model may output an indicator as to whether the speech is system-directed. The system may also incorporate other filters such as voice activity detection prior to speech recognition, or the like.

Detecting system-directed speech

Techniques for processing a user input are described. Text data representing a user input is processed with respect to at least one finite state transducer (FST) to generate at least one FST hypothesis. Context information may be required to traverse one or more paths of the at least one FST. The text data is also processed using at least one statistical model (e.g., perform intent classification, named entity recognition, and/or domain classification processing) to generate at least one statistical model hypothesis. The at least one FST hypothesis and the at least one statistical model hypothesis are input to a reranker that determines a most likely interpretation of the user input.

Natural language understanding processing

A system may include an acoustic event detection component for detecting acoustic events, which may be non-speech sounds. Upon detection of a command to detect a new sound, a device may prompt a user to cause occurrence of the sound one or more times. The acoustic event detection component may then be reconfigured, using audio data corresponding to the occurrences, to detect future occurrences of the event.

Acoustic event detection

Techniques for limiting natural language processing performed on input data are described. A system receives input data from a device. The input data corresponds to a command to be executed by the system. The system determines applications likely configured to execute the command. The system performs named entity recognition and intent classification with respect to only the applications likely configured to execute the command.

Natural language speech processing application selection

Growing community of inventors

Hopkinton, MA, United States of America

Spyridon Matsoukas