Amazon Technologies, Inc.

Raefer Christopher Gabriel

Arindam Mandal

Tagyoung Chung

Prakash Krishnan

Anushree Venkatesh

Ashish Kumar Agrawal

Suranjit Adhikari

Chandra Prakash Khatri

Behnam Hedayatnia

Devesh Mohan Pandey

Dilek Zeynep Hakkani-Tur

Rohit Prasad

Ariya Rastrow

Nikko Strom

Amitabh Saikia

Ankur Gandhe

Angeliki Metallinou

Siddhartha Reddy Jonnalagadda

Kemal Oral Cansizlar

Shishir Sridhar Bharathi

Rahul Goel

Savas Parastatidis

Govindarajan Sundaram Thattai

Anirudh Raju

David Chi-Wai Tang

Aishwarya Naresh Reganti

Abhay Kumar Jha

Qing Liu

Chien-wei Lin

Shucheng Zhu

Yun Lei

Arjit Biswas

Meyyappan Sundaram

Sanghyun Yi

Shanchan Wu

Ahmad Shikib Mehri

Sridhar Yadav Manoharan

Ankita De

Kjel Larsen

Techniques for facilitating natural language interactions with visual interactive content are described. During a build time, a system analyzes various websites and applications relating to a particular user goal to understand website and application navigation and information relating to the user goal. The learned information is used to store configuration data. During runtime, when a user request performance of an action, the system engages in a dialog with the user to complete the user's goal. The system uses the stored configuration data to determine actions to be performed at a website or application to complete the user's goal, and determines system responses to present to the user to facilitate completion of the goal. Such system responses may request information from the user, may inform the user of information displayed at the website or application, etc.

Natural language interactions with interactive visual content

Techniques for storing and using multi-session context are described. A system may store context data corresponding to a first interaction, where the context data may include action data, entity data and a profile identifier for a user. Later the stored context data may be retrieved during a second interaction corresponding to the entity of the second interaction. The second interaction may take place at a system different than the first interaction. The system may generate a response during the second interaction using the stored context data of the prior interaction.

Multi-session context

Techniques for conversation recovery in a dialog management system are described. A system may determine, using dialog models, that a predicted action to be performed by a skill component is likely to result in an undesired response or that the skill component is unable to respond to a user input of a dialog session. Rather than informing the user that the skill component is unable to respond, the system may send data to the skill component to enable the skill component to determine a correct action responsive to the user input. The data may include an indication of the predicted action and/or entity data corresponding to the user input. The system may receive, from the skill component, response data corresponding to the user input, and may use the response data to update a dialog context for the dialog session and an inference engine of the dialog management system.

Dialog management system

A dialog system receives a user request corresponding to a dialog with a user. The dialog system processes the user request to determine multiple service providers capable of responding to the user request. The dialog system selects one service provider based on a request-to-handle score, and selects another service provider based on a satisfaction rating. The dialog system updates the dialog state based on further input provided by the user to determine an output responsive to the user request.

Goal-oriented dialog system

A dialog manager receives text data corresponding to a dialog with a user. Entities represented in the text data are identified. Context data relating to the dialog is maintained, which may include prior dialog, prior API calls, user profile information, or other data. Using the text data and the context data, an N-best list of one or more dialog models is selected to process the text data. After processing the text data, the outputs of the N-best models are ranked and a top-scoring output is selected. The top-scoring output may be an API call and/or an audio prompt.

A system that can engage in a dialog with a user may select a system response to a user input based on how the system estimates a user may respond to a potential system response. Models may be trained to evaluate a potential system response in view of various available data including dialog history, entity data, etc. Each model may score the potential system response for various qualitative aspects such as whether the response is likely to be comprehensible, on-topic, interesting, likely to lead to the dialog continuing, etc. Such scores may be combined to other scores such as whether the potential response is coherent or engaging. The models may be trained using previous dialog/chatbot evaluation data. At runtime the scores may be used to select a system response to a user input as part of the dialog.

Dialog response generation

Described herein is a system for rescoring automatic speech recognition hypotheses for conversational devices that have multi-turn dialogs with a user. The system leverages dialog context by incorporating data related to past user utterances and data related to the system generated response corresponding to the past user utterance. Incorporation of this data improves recognition of a particular user utterance within the dialog.

Speech recognition using dialog history

Growing community of inventors

San Jose, CA, United States of America

Raefer Christopher Gabriel