Amazon Technologies, Inc.

Microsoft Technology Licensing, LLC

Rawles LLC

Nikko Strom

Rohit Prasad

Ariya Rastrow

Bjorn Hoffmeister

Arindam Mandal

Stephen Frederick Potter

Spyridon Matsoukas

Ryan Paul Thomas

Young-Bum Kim

Madan Mohan Rao Jampani

Angeliki Metallinou

Arushan Rajasekaram

Kenichi Kumatani

Ruhi Sarikaya

Shiva Kumar Sundaram

Yuzo Watanabe

Frédéric Johan Georges Deramat

Kerry Hammil

Minhua Wu

Vincent Auvray

Shiv Naga Prasad Vitaladevuni

Chenlei Guo

William Folwell Barton

Eli Joshua Fidler

Aaron A Eakin

Siddhartha Reddy Jonnalagadda

Sri Venkata Surya Siva Rama Krishna Garimella

Prakash Krishnan

Roland Maximilian Rolf Maas

Aaron Challenner

Diamond Bishop

Sanjeev Mishra

Minmin Shen

Dilek Zeynep Hakkani-Tur

Rohan Mutagi

Edward Dietz Crump

Tony David

Julian James Odell

Kavitha Velusamy

Hugh Evan Secker-Walker

Scott Ian Blanksteen

Stan Weidner Salvador

Oleg Rybakov

Ming Sun

Kenneth John Basye

Xing Fan

David Robert Thomas

Ying Shi

Tagyoung Chung

Sankaran Panchapagesan

Yixin Gao

Raefer Christopher Gabriel

Roberto Barra-Chicote

Karthik Ramakrishnan

Suranjit Adhikari

Isaac Joseph Madwed

Vasiliy Radostev

Julia Kennedy Nemer

Jon Hamaker

Anushree Venkatesh

Shuting Tang

Steven Mack Saunders

Casey Stuart Smith

Veerdhawal Pande

Alexandros Potamianos

Sungjin Lee

Sree Hari Krishnan Parthasarathi

Michael Maximilian Emanuel Bisani

Qing Liu

Zhenyu Yao

Thomas Edward Taylor

David Chi-Wai Tang

Lindo St Angel

Arijit Biswas

Xiaojiang Huang

Spyros Matsoukas

Sridhar Yadav Manoharan

Joo-Kyung Kim

Ankita De

Laura Maggia Panfili

Anna Caitlin Jentoft

Pablo Cesar Ganga

Pranav Prashant Ladkat

Josey Diego Sandoval

Nishtha Gupta

Bonan Zheng

Amotz Maimon

Yi Luo

Yohan Jo

David A Jara

Xinyan Zhao

Nikolaos Malandrakis

Jiahui Li

Jie Bao

Muddu Krishna Chintha

Priti Bisaria

Alexis Moinet

David Snyder

A system may be configured to receive and process various signals to generate a natural language description of a user's environment, called situational context data. The signals may include sensor data, device status, user activity, user input, and/or inferences made using such data. The situational context data may express a user-centric description of the user's environment; for example: 'User is taking a walk in the park on a sunny afternoon' or “activity: driving location: highway”, etc. The system may send the situational context data to various system components that may, for example, process speech, select applications/skills for handling user inputs, and/or that implement those applications/skills. The applications/skills may use the situational context data to provide recommendations, generate responses, and/or perform actions that are more relevant to the user's current environment.

Determining device context

Techniques for determining when speech is directed at another individual of a dialog, and storing a representation of such user-directed speech for use as context when processing subsequently-received system-directed speech are described. A system receives audio data and/or video data and determines therefrom that speech in the audio data is user-directed. Based on this, the system determine whether the speech is able to be used to perform an action by the system. If the speech is able to be used to perform an action, the system stores a natural language representation of the speech. Thereafter, when the system receives system-directed speech, the system generates a rewrite of a natural language representation of the system-directed speech based on the previously-received user-directed speech. The system then determines output data responsive to the system-directed speech using the rewritten natural language representation.

Natural language generation

A speech interface device is configured to detect an interrupt event and process a voice command without detecting a wakeword. The device includes on-device interrupt architecture configured to detect when device-directed speech is present and send audio data to a remote system for speech processing. This architecture includes an interrupt detector that detects an interrupt event (e.g., device-directed speech) with low latency, enabling the device to quickly lower a volume of output audio and/or perform other actions in response to a potential voice command. In addition, the architecture includes a device directed classifier that processes an entire utterance and corresponding semantic information and detects device-directed speech with high accuracy. Using the device directed classifier, the device may reject the interrupt event and increase a volume of the output audio or may accept the interrupt event, causing the output audio to end and performing speech processing on the audio data.

Device-directed utterance detection

A natural language system may be configured to act as a participant in a conversation between two users. The system may determine when a user expression such as speech, a gesture, or the like is directed from one user to the other. The system may processing input data related the expression (such as audio data, input data, language processing result data, conversation context data, etc.) to determine if the system should interject a response to the user-to-user expression. If so, the system may process the input data to determine a response and output it. The system may track that response as part of the data related to the ongoing conversation.

Dialog management for multiple users

A system that is capable of resolving anaphora using timing data received by a local device. A local device outputs audio representing a list of entries. The audio may represent synthesized speech of the list of entries. A user can interrupt the device to select an entry in the list, such as by saying 'that one.' The local device can determine an offset time representing the time between when audio playback began and when the user interrupted. The local device sends the offset time and audio data representing the utterance to a speech processing system which can then use the offset time and stored data to identify which entry on the list was most recently output by the local device when the user interrupted. The system can then resolve anaphora to match that entry and can perform additional processing based on the referred to item.

Exemplary embodiments provide distributed parallel training of a machine learning model. Multiple processors may be used to train a machine learning model to reduce training time. To synchronize trained model data between the processors, data is communicated between the processors after some number of training cycles. To improve the communication efficiency, exemplary embodiments synchronize data among a set of processors after a predetermined number of training cycles, and synchronize data between one or more processors of each set of the processors after a predetermined number of training cycles. During the first synchronization among a set of processors, compressed model gradient data generated after performing the training cycles may be communicated. During the second synchronization between the set of processors, trained models or full model gradient data generated after performing the training cycles may be communicated.

Distributed model training

A speech interface device is configured to receive response data from a remote speech processing system for responding to user speech. This response data may be enhanced with information such as remote NLU data. The response data from the remote speech processing system may be compared to local NLU data to improve a speech processing model on the device. Thus, the device may perform supervised on-device learning based on the remote NLU data. The device may determine differences between the updated speech processing model and an original speech processing model received from the remote system and may send data indicating these differences to the remote system. The remote system may aggregate data received from a plurality of devices and may generate an improved speech processing model.

On-device learning in a hybrid speech processing system

Techniques for speech processing using a deep neural network (DNN) based acoustic model front-end are described. A new modeling approach directly models multi-channel audio data received from a microphone array using a first model (e.g., multi-geometry/multi-channel DNN) that is trained using a plurality of microphone array geometries. Thus, the first model may receive a variable number of microphone channels, generate multiple outputs using multiple microphone array geometries, and select the best output as a first feature vector that may be used similarly to beamformed features generated by an acoustic beamformer. A second model (e.g., feature extraction DNN) processes the first feature vector and transforms it to a second feature vector having a lower dimensional representation. A third model (e.g., classification DNN) processes the second feature vector to perform acoustic unit classification and generate text data. The DNN front-end enables improved performance despite a reduction in microphones.

Deep multi-channel acoustic modeling using multiple microphone array geometries

Described herein is a system for responding to a frustrated user with a response determined based on spoken language understanding (SLU) processing of a user input. The system detects user frustration and responds to a repeated user input by confirming an action to be performed or presenting an alternative action, instead of performing the action responsive to the user input. The system also detects poor audio quality of the captured user input, and responds by requesting the user to repeat the user input. The system processes sentiment data and signal quality data to respond to user inputs.

Sentiment aware voice user interface

Techniques for speech processing using a deep neural network (DNN) based acoustic model front-end are described. A new modeling approach directly models multi-channel audio data received from a microphone array using a first model (e.g., multi-channel DNN) that takes in raw signals and produces a first feature vector that may be used similarly to beamformed features generated by an acoustic beamformer. A second model (e.g., feature extraction DNN) processes the first feature vector and transforms it to a second feature vector having a lower dimensional representation. A third model (e.g., classification DNN) processes the second feature vector to perform acoustic unit classification and generate text data. These three models may be jointly optimized for speech processing (as opposed to individually optimized for signal enhancement), enabling improved performance despite a reduction in microphones and a reduction in bandwidth consumption during real-time processing.

Deep multi-channel acoustic modeling

Techniques for limiting natural language processing performed on input data are described. A system receives input data from a device. The input data corresponds to a command to be executed by the system. The system determines applications likely configured to execute the command. The system performs named entity recognition and intent classification with respect to only the applications likely configured to execute the command.

Natural language speech processing application selection

A dialog manager receives text data corresponding to a dialog with a user. Entities represented in the text data are identified. Context data relating to the dialog is maintained, which may include prior dialog, prior API calls, user profile information, or other data. Using the text data and the context data, an N-best list of one or more dialog models is selected to process the text data. After processing the text data, the outputs of the N-best models are ranked and a top-scoring output is selected. The top-scoring output may be an API call and/or an audio prompt.

Goal-oriented dialog system

Features are disclosed for generating predictive personal natural language processing models based on user-specific profile information. The predictive personal models can provide broader coverage of the various terms, named entities, and/or intents of an utterance by the user than a personal model, while providing better accuracy than a general model. Profile information may be obtained from various data sources. Predictions regarding the content or subject of future user utterances may be made from the profile information. Predictive personal models may be generated based on the predictions. Future user utterances may be processed using the predictive personal models.

Generation of predictive natural language processing models

Growing community of inventors

Kirkland, WA, United States of America

Nikko Strom