Amazon Technologies, Inc.

Ying Shi

Lambert Leo Mathias

Frederic Johan Georges Deramat

Ryan Paul Thomas

Imre Attila Kiss

Angeliki Metallinou

Weam Abu Zaki

Arijit Biswas

Giuseppe Di Fabbrizio

Nikko Strom

Rajiv Ramachandran

Shishir Sridhar Bharathi

Aaron Challenner

Anushree Venkatesh

Veerdhawal Pande

Vincent Auvray

Alexandros Potamianos

Hann Wang

Bonan Zheng

Yohan Jo

David A Jara

Xinyan Zhao

Nikolaos Malandrakis

Jiahui Li

Techniques for generating a prompt for a language model to determine an action responsive to a user input, are described. In some embodiments, the system receives a user input, determines one or more application programming interfaces (APIs) configured to perform actions that are relevant to the user input and exemplars representing examples of using the APIs with respect to user inputs similar to the current user input. The system further determines device states of devices that are determined to be related to the user input and also determines other contextual information (e.g., weather information, time of day, geographic location, etc.). The system generates a prompt including the user input, the APIs, the exemplars, the device states, and the other contextual information. A language model processes the prompt to determine an action responsive to the user input and the system causes performance of the action.

Natural language generation

Techniques for determining when speech is directed at another individual of a dialog, and storing a representation of such user-directed speech for use as context when processing subsequently-received system-directed speech are described. A system receives audio data and/or video data and determines therefrom that speech in the audio data is user-directed. Based on this, the system determine whether the speech is able to be used to perform an action by the system. If the speech is able to be used to perform an action, the system stores a natural language representation of the speech. Thereafter, when the system receives system-directed speech, the system generates a rewrite of a natural language representation of the system-directed speech based on the previously-received user-directed speech. The system then determines output data responsive to the system-directed speech using the rewritten natural language representation.

Features are disclosed for processing a user utterance with respect to multiple subject matters or domains, and for selecting a likely result from a particular domain with which to respond to the utterance or otherwise take action. A user utterance may be transcribed by an automatic speech recognition ('ASR') module, and the results may be provided to a multi-domain natural language understanding ('NLU') engine. The multi-domain NLU engine may process the transcription(s) in multiple individual domains rather than in a single domain. In some cases, the transcription(s) may be processed in multiple individual domains in parallel or substantially simultaneously. In addition, hints may be generated based on previous user interactions and other data. The ASR module, multi-domain NLU engine, and other components of a spoken language processing system may use the hints to more efficiently process input or more accurately generate output.

Architecture for multi-domain natural language processing

Features are disclosed for processing and interpreting natural language, such as interpretations of user utterances, in multi-turn dialog interactions. Context information regarding interpretations of user utterances and system responses to the user utterances can be maintained. Subsequent user utterances can be interpreted using the context information, rather than being interpreted without context. In some cases, interpretations of subsequent user utterances can be merged with interpretations of prior user utterances using a rule-based framework. Rules may be defined to determine which interpretations may be merged and under what circumstances they may be merged.

Context interpretation in natural language processing using previous dialog acts

Features are disclosed for determining a definition or value of a nonstandard term. A user utterance may be processed into one or more candidate transcriptions. An interpretation of the utterance can be generated from the transcriptions. If the transcription includes a word, phrase, or term that is not recognized or is used in a nonstandard way, one or more data stores may be queried regarding the proper value or definition of the term. If a definition or value is not available in the data stores, the user may be prompted to provide one. The user-supplied definition can be saved for future use, and may be used as a general definition of the term for other users.

Interpreting nonstandard terms in language processing using text-based communications

Features are disclosed for determining an element of a user utterance or user intent in conjunction with one or more related elements of the user utterance or user intent. A user utterance may be transcribed by an automatic speech recognition ('ASR') module, and the results may be provided to a natural language understanding ('NLU') module. The NLU module may perform named entity recognition, intent classification, and/or other processes on the ASR results. In addition, the NLU module may determine or verify the values associated with the recognized named entities using a data store of known values. When two or more named entities are related, their values may be determined and/or verified in conjunction with each other in order to preserve the relationship between them.

Named entity resolution in spoken language processing

Architecture for multi-domain utterance processing

Growing community of inventors

Seattle, WA, United States of America

Ying Shi