Amazon Technologies, Inc.

Xing Fan

Chenlei Guo

Ruhi Sarikaya

Anjishnu Kumar

Arpit Gupta

Kai Wei

Shuting Tang

Narendra Gyanchandani

Chengyuan Ma

Puyang Xu

Hyungseo Park

Gokhan Tur

Ariya Rastrow

Bjorn Hoffmeister

Spyridon Matsoukas

Saurabh Gupta

Xiaohu Liu

Nikko Strom

Lambert Leo Mathias

Bharath Bhimanaik Kumar

Ankur Gandhe

Hung The Pham

Shubham Katiyar

Aaron A Eakin

Kerry Hammil

Vasiliy Radostev

Xinhong Zhang

Yuzong Liu

Sixing Lu

Casey Stuart Smith

Angela Sun

Xiaojiang Huang

Jin Hock Ong

Tongfei Chen

Yiming Wang

I-Fan Chen

Eunah Cho

Yi Luo

Yuan Ling

Zheng Chen

Dinesh Malla

Jie Bao

Muddu Krishna Chintha

Priti Bisaria

Robert O'Neal Copeland

Devices and techniques are generally described for arbitration between LLM-based and intent-based natural language processing flows. In various examples, an automatic speech recognition (ASR) component may generate first ASR output data representing the first natural language input. A first machine learning model may select an LLM-based natural language processing flow. The first machine learning model may be trained to select between at least the LLM-based and non-LLM-based natural language processing flows. The first ASR output data may be processed using the LLM-based processing flow. The LLM-based processing flow may generate first executable data.

Language model arbitration for natural language processing

A system may be configured to receive and process various signals to generate a natural language description of a user's environment, called situational context data. The signals may include sensor data, device status, user activity, user input, and/or inferences made using such data. The situational context data may express a user-centric description of the user's environment; for example: 'User is taking a walk in the park on a sunny afternoon' or “activity: driving location: highway”, etc. The system may send the situational context data to various system components that may, for example, process speech, select applications/skills for handling user inputs, and/or that implement those applications/skills. The applications/skills may use the situational context data to provide recommendations, generate responses, and/or perform actions that are more relevant to the user's current environment.

Determining device context

Techniques for generating tasks to be completed in order to perform an action responsive to a user input and, for a given task, shortlisting available components to those that are relevant for the task are described. The system processes a user input to determine tasks to be completed in order to perform an action responsive to the user input. The system determines a priority of the tasks and selects a top-ranked task. The system determines descriptions of processing performable by components that are semantically similar to the current task, and requests a description of the function the corresponding components would perform for the current task. Based on the received descriptions, the system selects one or more components to perform the task. Thereafter, the system causes the action to be performed and outputs a response to the user input.

Natural language processing

Techniques for determining one or more responses associated with one or more components that are responsive to a user input are described. The system receives a user input and causes one or more components to generate one or more responses associated with the user input. The system determines one or more of the responses are responsive to the user input, causes one or more actions associated with the responses to be performed, and outputs a natural language summary of the one or more responses. If the system determines that none of the responses are responsive to the user input and/or an ambiguity exists with respect to the user input, the system can generate a request for additional information usable to resolve the ambiguity, which may be sent to another component of the system and/or output to the user that provided the user input.

Techniques for processing with respect to a user input as contextual information is available are described. A system generates a first task prediction using first context data that is available when a user input is received. The system generates a second task prediction (e.g., updated first task prediction) when second context data is received, and then further generates a third task prediction when third context data is received. Example first context data may include device type information, time information, location, etc. Example second context data may include automatic speech recognition (ASR) data. Example third context data may include natural language understanding (NLU) data. Using the third task prediction, the system generates an output responsive to the user input.

Natural language processing system

A system is provided for a self-learning policy engine that can be used by various spoken language understanding (SLU) processing components. The system also provides for sharing contextual information from processing performed by an upstream SLU component to a downstream SLU component to facilitate decision making by the downstream SLU component. The system also provides for a SLU component to select from a variety of actions to take. A SLU component may implement an instance of the self-learning policy that is specifically configured for the particular SLU component.

Spoken language understanding system

Techniques for determining a command or intent likely to be subsequently invoked by a user of a system are described. A user inputs a command (either via a spoken utterance or textual input) to a system. The system determines content responsive to the command. The system also determines a second command or corresponding intent likely to be invoked by the user subsequent to the previous command. Such determination may involve analyzing pairs of intents, with each pair being associated with a probability that one intent of the pair will be invoked by a user subsequent to a second intent of the pair. The system then outputs first content responsive to the first command and second content soliciting the user as to whether the system to execute the second command.

Proactive command framework

Techniques for determining and using interaction affinity data are described. Interaction affinity data may indicate a latent affinity between information corresponding to an interaction, such as, intents, entities, device type from which a user input is received, domain, etc. A system may use the interaction affinity data to determine an alternative input representation for a spoken input to cause output of a desired response to the spoken input. The system may also use the interaction affinity data to recommend an action to a user.

Interaction data and processing natural language inputs

Techniques for determining an intent for a user input in a dialog are described. The system processes historic interaction data that is structured based skills and intents, with each skill-intent pair being associated with one or more past user inputs received by the system, one or more sample inputs, and one or more alternative representations of the user inputs. Based on processing of the historic interaction data and dialog data of previous turns of the dialog, the system determines potential intents for the user input of the current turn of the dialog. The potential intents may correspond to a presently active skill or another skill, enabling the user to interact with another skill during the dialog.

Dialog management system

Techniques for determining an intent of a subsequent user input in a dialog are described. The system processes historic interaction data that is structured based on natural language understanding (NLU) hypotheses, with each NLU hypothesis being associated with one or more past user inputs received by the system, one or more sample inputs, and one or more past system responses. Based on processing of the historic interaction data and dialog data of previous turns of the dialog, the system determines candidate intents for the subsequent turn of the dialog. The system also uses context data to determine the candidate intents.

Techniques for handling errors during processing of natural language inputs are described. A system may process a natural language input to generate an ASR hypothesis or NLU hypothesis. The system may use more than one data searching technique (e.g., deep neural network searching, convolutional neural network searching, etc.) to generate an alternate ASR hypothesis or NLU hypothesis, depending on the type of hypothesis input for alternate hypothesis processing.

Alternate natural language input generation

A system is provided for reducing friction during user interactions with a natural language processing system, such as voice assistant systems. The system determines a pre-trained model using dialog session data corresponding to multiple user profiles. The system determines a fine-tuned model using the pre-trained model and a fine-tuning dataset that corresponds to a particular task, such as query rewriting. The system uses the fine-tuned model to process a user input and determine an alternative representation of the input that can result in a desired response from the natural language processing system.

Natural language understanding

Growing community of inventors

Redmond, WA, United States of America

Xing Fan