Amazon Technologies, Inc.

Arindam Mandal

Spyridon Matsoukas

Shiva Kumar Sundaram

Raefer Christopher Gabriel

Aparna Khare

Shiv Naga Prasad Vitaladevuni

Nikko Strom

Kenichi Kumatani

Minhua Wu

Rohit Prasad

Angeliki Metallinou

Prakash Krishnan

Tagyoung Chung

Anirudh Raju

Ariya Rastrow

Bjorn Hoffmeister

Sankaran Panchapagesan

Sundararajan Srinivasan

Chao Wang

Siddhartha Reddy Jonnalagadda

Krishna Subramanian

Anushree Venkatesh

Vishwanathan Krishnamoorthy

Dilek Zeynep Hakkani-Tur

Yang Liu

Ashish Kumar Agrawal

Yue Liu

Ming Sun

James David Meyers

Ankur Gandhe

Shamitha Somashekar

Premkumar Natarajan

Chandra Prakash Khatri

Suranjit Adhikari

Jing Huang

Behnam Hedayatnia

Jérémie Lecomte

Shah Samir Pravinchandra

Vincent Auvray

David Chi-Wai Tang

Anjali Yuan Narayan-Chen

Shereen Oraby

Minmin Shen

Jiao Sun

Ahmad Shikib Mehri

Chien-wei Lin

Devesh Mohan Pandey

Gregory Michael Hart

Daniel N Miller

Lambert Leo Mathias

Ryan Paul Thomas

Amitabh Saikia

Ying Shi

Kemal Oral Cansizlar

Aaron Challenner

Shishir Sridhar Bharathi

Thomas Kollar

Govindarajan Sundaram Thattai

Savas Parastatidis

Sai Sailesh Kopuri

Aishwarya Naresh Reganti

Thomas Edward Taylor

Abhay Kumar Jha

Arlen R Dean

Qing Liu

Anish Acharya

John Moore

Nanyun Peng

Alessandra Cervone

Shachi Paul

Nanyung Peng

Shubhra Chandra

Shuyang Gao

Shanchan Wu

Meyyappan Sundaram

Arjit Biswas

Yun Lei

Daniel Miller

Arlen Dean

Amotz Maimon

Sridhar Yadav Manoharan

Ankita De

Bonan Zheng

Nishtha Gupta

Josey Diego Sandoval

Kjel Larsen

Shucheng Zhu

Described herein is a system for rescoring automatic speech recognition hypotheses for conversational devices that have multi-turn dialogs with a user. The system leverages dialog context by incorporating data related to past user utterances and data related to the system generated response corresponding to the past user utterance. Incorporation of this data improves recognition of a particular user utterance within the dialog.

Speech recognition using dialog history

In accordance with one disclosed method, first data representing first text may be processed using a first machine learning model configured to generate second data representing an explanation as to why the first text is humorous. The second data may be processed, together with the first data, by a second machine learning model configured to generate a value indicating that the first data and the second data correspond to a pun. A device may be caused to take at least a first action based at least in part on the value.

Virtual assistant humor management

In accordance with one disclosed method, first data representing one or more first keywords may be processed by at least one first component to determine a first pair of words suitable for generating a pun. The first pair of words may then be processed using a first machine learning model configured to generate an output pun based at least in part on one or more input keywords and an input word pair such that the output pun is contextually related to the one or more input keywords. A first pun that is contextually related to the one or more first keywords may be received from the first machine learning model, and a first device may be caused to output a representation of the first pun.

Systems and methods for utilizing microphone array information for acoustic modeling are disclosed. Audio data may be received from a device having a microphone array configuration. Microphone configuration data may also be received that indicates the configuration of the microphone array. The microphone configuration data may be utilized as an input vector to an acoustic model, along with the audio data, to generate phoneme data. Additionally, the microphone configuration data may be utilized to train and/or generate acoustic models, select an acoustic model to perform speech recognition with, and/or to improve trigger sound detection.

Speech processing optimizations based on microphone array

Techniques for facilitating natural language interactions with visual interactive content are described. During a build time, a system analyzes various websites and applications relating to a particular user goal to understand website and application navigation and information relating to the user goal. The learned information is used to store configuration data. During runtime, when a user request performance of an action, the system engages in a dialog with the user to complete the user's goal. The system uses the stored configuration data to determine actions to be performed at a website or application to complete the user's goal, and determines system responses to present to the user to facilitate completion of the goal. Such system responses may request information from the user, may inform the user of information displayed at the website or application, etc.

Natural language interactions with interactive visual content

A natural language system may be configured to act as a participant in a conversation between two users. The system may determine when a user expression such as speech, a gesture, or the like is directed from one user to the other. The system may processing input data related the expression (such as audio data, input data, language processing result data, conversation context data, etc.) to determine if the system should interject a response to the user-to-user expression. If so, the system may process the input data to determine a response and output it. The system may track that response as part of the data related to the ongoing conversation.

Dialog management for multiple users

A system may use multiple speech interface devices to interact with a user by speech. All or a portion of the speech interface devices may detect a user utterance and may initiate speech processing to determine a meaning or intent of the utterance. Within the speech processing, arbitration is employed to select one of the multiple speech interface devices to respond to the user utterance. Arbitration may be based in part on metadata that directly or indirectly indicates the proximity of the user to the devices, and the device that is deemed to be nearest the user may be selected to respond to the user utterance.

Device selection for providing a response

A system that is capable of resolving anaphora using timing data received by a local device. A local device outputs audio representing a list of entries. The audio may represent synthesized speech of the list of entries. A user can interrupt the device to select an entry in the list, such as by saying 'that one.' The local device can determine an offset time representing the time between when audio playback began and when the user interrupted. The local device sends the offset time and audio data representing the utterance to a speech processing system which can then use the offset time and stored data to identify which entry on the list was most recently output by the local device when the user interrupted. The system can then resolve anaphora to match that entry and can perform additional processing based on the referred to item.

Techniques for storing and using multi-session context are described. A system may store context data corresponding to a first interaction, where the context data may include action data, entity data and a profile identifier for a user. Later the stored context data may be retrieved during a second interaction corresponding to the entity of the second interaction. The second interaction may take place at a system different than the first interaction. The system may generate a response during the second interaction using the stored context data of the prior interaction.

Multi-session context

Techniques for enrolling a user in a system's user recognition functionality without requiring the user speak particular speech are described. The system may determine characteristics unique to a user input. The system may generate an implicit voice profile from user inputs having similar characteristics. After an implicit voice profile is generated, the system may receive a user input having speech characteristics similar to that of the implicit voice profile. The system may ask the user if the user wants the system to associate the implicit voice profile with a particular user identifier. If the user responds affirmatively, the system may request an identifier of a user profile (e.g., a user name). In response to receiving the user's name, the system may identify a user profile associated with the name and associate the implicit voice profile with the user profile, thereby converting the implicit voice profile into an explicit voice profile.

Speech based user recognition

Techniques for conversation recovery in a dialog management system are described. A system may determine, using dialog models, that a predicted action to be performed by a skill component is likely to result in an undesired response or that the skill component is unable to respond to a user input of a dialog session. Rather than informing the user that the skill component is unable to respond, the system may send data to the skill component to enable the skill component to determine a correct action responsive to the user input. The data may include an indication of the predicted action and/or entity data corresponding to the user input. The system may receive, from the skill component, response data corresponding to the user input, and may use the response data to update a dialog context for the dialog session and an inference engine of the dialog management system.

Dialog management system

A dialog system receives a user request corresponding to a dialog with a user. The dialog system processes the user request to determine multiple service providers capable of responding to the user request. The dialog system selects one service provider based on a request-to-handle score, and selects another service provider based on a satisfaction rating. The dialog system updates the dialog state based on further input provided by the user to determine an output responsive to the user request.

Goal-oriented dialog system

A speech-capture device can capture audio data during wakeword monitoring and use the audio data to determine if a user is present nearby the device, even if no wakeword is spoken. Audio such as speech, human originating sounds (e.g., coughing, sneezing), or other human related noises (e.g., footsteps, doors closing) can be used to detect audio. Audio frames are individually scored as to whether a human presence is detected in the particular audio frames. The scores are then smoothed relative to nearby frames to create a decision for a particular frame. Presence information can then be sent according to a periodic schedule to a remote device to create a presence 'heartbeat' that regularly identifies whether a user is detected proximate to a speech-capture device.

User presence detection

Techniques for speech processing using a deep neural network (DNN) based acoustic model front-end are described. A new modeling approach directly models multi-channel audio data received from a microphone array using a first model (e.g., multi-channel DNN) that takes in raw signals and produces a first feature vector that may be used similarly to beamformed features generated by an acoustic beamformer. A second model (e.g., feature extraction DNN) processes the first feature vector and transforms it to a second feature vector having a lower dimensional representation. A third model (e.g., classification DNN) processes the second feature vector to perform acoustic unit classification and generate text data. These three models may be jointly optimized for speech processing (as opposed to individually optimized for signal enhancement), enabling improved performance despite a reduction in microphones and a reduction in bandwidth consumption during real-time processing.

Deep multi-channel acoustic modeling

Growing community of inventors

Redwood City, CA, United States of America

Arindam Mandal