Amazon Technologies, Inc.

Angeliki Metallinou

Arindam Mandal

Nikko Strom

Rahul Goel

Vincent Auvray

Dilek Zeynep Hakkani-Tur

Rohit Prasad

Ariya Rastrow

Siddhartha Reddy Jonnalagadda

Prakash Krishnan

Tagyoung Chung

Aaron Challenner

Vishal Ishwar

Anish Acharya

Minmin Shen

Arijit Biswas

Shiv Naga Prasad Vitaladevuni

Lambert Leo Mathias

Ying Shi

Raefer Christopher Gabriel

Suranjit Adhikari

Thomas Kollar

Anushree Venkatesh

Inderjit Dhillon

Alexandros Potamianos

Veerdhawal Pande

Qing Liu

Chien-wei Lin

Thomas Edward Taylor

David Chi-Wai Tang

Jiahui Li

Shachi Paul

Nikolaos Malandrakis

Xinyan Zhao

David A Jara

Yohan Jo

Bonan Zheng

Hann Wang

Amotz Maimon

Sridhar Yadav Manoharan

Nishtha Gupta

Josey Diego Sandoval

Ankita De

Shubhra Chandra

Techniques for generating a prompt for a language model to determine an action responsive to a user input, are described. In some embodiments, the system receives a user input, determines one or more application programming interfaces (APIs) configured to perform actions that are relevant to the user input and exemplars representing examples of using the APIs with respect to user inputs similar to the current user input. The system further determines device states of devices that are determined to be related to the user input and also determines other contextual information (e.g., weather information, time of day, geographic location, etc.). The system generates a prompt including the user input, the APIs, the exemplars, the device states, and the other contextual information. A language model processes the prompt to determine an action responsive to the user input and the system causes performance of the action.

Natural language generation

Techniques for determining when speech is directed at another individual of a dialog, and storing a representation of such user-directed speech for use as context when processing subsequently-received system-directed speech are described. A system receives audio data and/or video data and determines therefrom that speech in the audio data is user-directed. Based on this, the system determine whether the speech is able to be used to perform an action by the system. If the speech is able to be used to perform an action, the system stores a natural language representation of the speech. Thereafter, when the system receives system-directed speech, the system generates a rewrite of a natural language representation of the system-directed speech based on the previously-received user-directed speech. The system then determines output data responsive to the system-directed speech using the rewritten natural language representation.

A natural language system may be configured to act as a participant in a conversation between two users. The system may determine when a user expression such as speech, a gesture, or the like is directed from one user to the other. The system may processing input data related the expression (such as audio data, input data, language processing result data, conversation context data, etc.) to determine if the system should interject a response to the user-to-user expression. If so, the system may process the input data to determine a response and output it. The system may track that response as part of the data related to the ongoing conversation.

Dialog management for multiple users

A system that is capable of resolving anaphora using timing data received by a local device. A local device outputs audio representing a list of entries. The audio may represent synthesized speech of the list of entries. A user can interrupt the device to select an entry in the list, such as by saying 'that one.' The local device can determine an offset time representing the time between when audio playback began and when the user interrupted. The local device sends the offset time and audio data representing the utterance to a speech processing system which can then use the offset time and stored data to identify which entry on the list was most recently output by the local device when the user interrupted. The system can then resolve anaphora to match that entry and can perform additional processing based on the referred to item.

Multi-modal natural language processing systems are provided. Some systems are context-aware systems that use multi-modal data to improve the accuracy of natural language understanding as it is applied to spoken language input. Machine learning architectures are provided that jointly model spoken language input ('utterances') and information displayed on a visual display ('on-screen information'). Such machine learning architectures can improve upon, and solve problems inherent in, existing spoken language understanding systems that operate in multi-modal contexts.

Natural language processing with contextual data representing displayed content

A dialog generator receives data corresponding to desired dialog, such as application programming interface (API) information and sample dialog. A first model corresponding to an agent simulator and a second model corresponding to a user simulator take turns creating a plurality of dialog outlines of the desired dialog. The dialog generator may determine that one or more additional APIs are relevant to the dialog and may create further dialog outlines related thereto. The dialog outlines are converted to natural dialog to generate the dialog.

Goal-oriented dialog generation using dialog template, API, and entity data

A dialog manager receives text data corresponding to a dialog with a user. Entities represented in the text data are identified. Context data relating to the dialog is maintained, which may include prior dialog, prior API calls, user profile information, or other data. Using the text data and the context data, an N-best list of one or more dialog models is selected to process the text data. After processing the text data, the outputs of the N-best models are ranked and a top-scoring output is selected. The top-scoring output may be an API call and/or an audio prompt.

Goal-oriented dialog system

A natural language understanding (NLU) system that uses a reduced dimensionality of word embedding features to configure compressed NLU models that use reduced computing resources for NLU tasks. A modified NLU model may include a compressed vocabulary data structure of word embedding data vectors that include a set of values corresponding to a reduced dimensionality of the original word embedding features, resulting in a smaller sized vocabulary data structure and reduced size of the vocabulary data structure. Further components of the modified NLU model perform matrix operations to expand the dimensionality of the reduced word embedding data vectors up to the expected dimensionality of later layers of the NLU model. Additional training and reweighting can adjust for potential loses in performance resulting from reductions in the word embedding features. Thus the modified NLU model can achieve similar performance to an original NLU model with reductions in use of computing resources.

Natural language processing

Multi-modal natural language processing

A system capable of performing natural language understanding (NLU) without the concept of a domain that influences NLU results. The present system uses a hierarchical organizations of intents/commands and entity types, and trained models associated with those hierarchies, so that commands and entity types may be determined for incoming text queries without necessarily determining a domain for the incoming text. The system thus operates in a domain agnostic manner, in a departure from multi-domain architecture NLU processing where a system determines NLU results for multiple domains simultaneously and then ranks them to determine which to select as the result.

Fine-grained natural language understanding

Growing community of inventors

Mountain View, CA, United States of America

Angeliki Metallinou