Amazon Technologies, Inc.

Microsoft Technology Licensing, LLC

Other

Chenlei Guo

Xing Fan

Alireza Roshan Ghias

Ye-Yi Wang

Chengyuan Ma

Shuting Tang

Pragaash Ponnusamy

Jianfeng Gao

Xiaohu Liu

Divya Jetley

Byungki Byun

Kai Wei

Sixing Lu

Bigyan Rajbhandari

Xiaodong He

Nikko Strom

Xinying Song

Brian D Remick

Narendra Gyanchandani

Abhinav Sethy

Yelong Shen

Praveen Kumar Bodigutla

Zhenyu Yao

Hyungseo Park

Puyang Xu

Stephen Friesen

Zhenxiang Zhou

Karen Catelyn Stabile

Kevin Quinn

Edward Thiele

Mohammed Aatif Ali

Marcus Gois

Li Deng

Gokhan Tur

Ruhi Sarikaya

Kai Chieh Liu

Ariya Rastrow

Spyridon Matsoukas

Saurabh Gupta

Chinna Babu Polinati

Lambert Leo Mathias

Young-Bum Kim

Bharath Bhimanaik Kumar

David Robert Thomas

Ankur Gandhe

Jianshu Chen

Joe Pemberton

Shubham Katiyar

Aaron A Eakin

Hung The Pham

Kerry Hammil

Isaac Joseph Madwed

Vasiliy Radostev

Julia Kennedy Nemer

Xinhong Zhang

Casey Stuart Smith

Pravjit Tiwana

Steven Mack Saunders

Sungjin Lee

Xiaojiang Huang

Zheng Du

Hung Tuan Pham

Angela Sun

Mariana Stepp

Jie Bao

Dinesh Malla

Zheng Chen

Yuan Ling

Muddu Krishna Chintha

Priti Bisaria

Zhongkai Sun

Yi Luo

Pavel Metrikov

Sriharsha Venkata Chintalapati

Robert O'Neal Copeland

Fangfu Li

Sean William Jewell

Eunah Cho

Pablo Cesar Ganga

Anna Caitlin Jentoft

Laura Maggia Panfili

Joo-Kyung Kim

Jin Hock Ong

Clint Solomon Mathialagan

Christopher Ball

Yang Zou

Techniques for determining alternative input representations using entity expansion and entity weighting are described. An entity expansion knowledge base is built by extracting entities from user input-system response pairs that resulted in satisfactory experiences. An extracted entity is associated with an initial score based on it being included in the user input only, in the system response only or both the user input and the system response. Entities co-occurring in the user input-system response pair are connected in the knowledge base. An overall score is associated with the connections based on the initial scores of the connected entities. Using the knowledge base, expansion entities related to an entity included in a user input are determined, and the expansion entities and user input entity are weighted. The weighting of the entities involves assigning a level to each entity based on pairs of user input-alternative input representation.

Alternative input representations

Devices and techniques are generally described for arbitration between LLM-based and intent-based natural language processing flows. In various examples, an automatic speech recognition (ASR) component may generate first ASR output data representing the first natural language input. A first machine learning model may select an LLM-based natural language processing flow. The first machine learning model may be trained to select between at least the LLM-based and non-LLM-based natural language processing flows. The first ASR output data may be processed using the LLM-based processing flow. The LLM-based processing flow may generate first executable data.

Language model arbitration for natural language processing

A system may be configured to receive and process various signals to generate a natural language description of a user's environment, called situational context data. The signals may include sensor data, device status, user activity, user input, and/or inferences made using such data. The situational context data may express a user-centric description of the user's environment; for example: 'User is taking a walk in the park on a sunny afternoon' or “activity: driving location: highway”, etc. The system may send the situational context data to various system components that may, for example, process speech, select applications/skills for handling user inputs, and/or that implement those applications/skills. The applications/skills may use the situational context data to provide recommendations, generate responses, and/or perform actions that are more relevant to the user's current environment.

Determining device context

Techniques for generating tasks to be completed in order to perform an action responsive to a user input and, for a given task, shortlisting available components to those that are relevant for the task are described. The system processes a user input to determine tasks to be completed in order to perform an action responsive to the user input. The system determines a priority of the tasks and selects a top-ranked task. The system determines descriptions of processing performable by components that are semantically similar to the current task, and requests a description of the function the corresponding components would perform for the current task. Based on the received descriptions, the system selects one or more components to perform the task. Thereafter, the system causes the action to be performed and outputs a response to the user input.

Natural language processing

Techniques for determining one or more responses associated with one or more components that are responsive to a user input are described. The system receives a user input and causes one or more components to generate one or more responses associated with the user input. The system determines one or more of the responses are responsive to the user input, causes one or more actions associated with the responses to be performed, and outputs a natural language summary of the one or more responses. If the system determines that none of the responses are responsive to the user input and/or an ambiguity exists with respect to the user input, the system can generate a request for additional information usable to resolve the ambiguity, which may be sent to another component of the system and/or output to the user that provided the user input.

Techniques for processing with respect to a user input as contextual information is available are described. A system generates a first task prediction using first context data that is available when a user input is received. The system generates a second task prediction (e.g., updated first task prediction) when second context data is received, and then further generates a third task prediction when third context data is received. Example first context data may include device type information, time information, location, etc. Example second context data may include automatic speech recognition (ASR) data. Example third context data may include natural language understanding (NLU) data. Using the third task prediction, the system generates an output responsive to the user input.

Natural language processing system

A system is provided for a self-learning policy engine that can be used by various spoken language understanding (SLU) processing components. The system also provides for sharing contextual information from processing performed by an upstream SLU component to a downstream SLU component to facilitate decision making by the downstream SLU component. The system also provides for a SLU component to select from a variety of actions to take. A SLU component may implement an instance of the self-learning policy that is specifically configured for the particular SLU component.

Spoken language understanding system

Techniques for performing centralized unsuperivised learning in a multi-domain system are described. A user may request labeled data for an ML task, where the request includes a prompt for obtaining relevant explicit user feedback. The system may use the prompt to collect explicit user feedback for relevant runtime user inputs. After a duration of time (in the user's request for labeled data) has elapsed, the system determines whether collected user feedback indicates processing of the user input was defective and, if so, determines a cause of the defective processing. The system then uses one or more label generators to generate labeled data using the collected user feedback, whether the processing was defective, and the determined defect cause.

Machine learning label generation

Techniques for presenting an alternative input representation to a user for testing and collecting processing data are described. A system may determine that a received spoken input triggers an alternative input representation for presenting. The system may output data corresponding to the alternative input representation in response to the received spoken input, and the system may receive user feedback from the user. The system may store the user feedback and processing data corresponding to processing of the alternative input representation, which may be later used to update an alternative input component configured to determine alternative input representations for spoken inputs.

Alternative input representations for speech inputs

Techniques for decreasing (or eliminating) the possibility of a skill performing an action that is not responsive to a corresponding user input are described. A system may train one or more machine learning models with respect to user inputs, which resulted in incorrect actions being performed by skills, and corresponding user inputs, which resulted in the correct action being performed. The system may use the trained machine learning model(s) to rewrite user inputs that, if not rewritten, may result in incorrect actions being performed. The system may implement the trained machine learning model(s) with respect to ASR output text data to determine if the ASR output text data corresponds (or substantially corresponds) to previous ASR output text data that resulted in an incorrect action being performed. If the trained machine learning model(s) indicates the present ASR output text data corresponds (or substantially corresponds) to such previous ASR output text data, the system may rewrite the present ASR output text data to correspond to text data representing a rephrase of the user input that will (or is more likely to) result in a correct action being performed.

Learning how to rewrite user-specific input for natural language understanding

Techniques for determining and using interaction affinity data are described. Interaction affinity data may indicate a latent affinity between information corresponding to an interaction, such as, intents, entities, device type from which a user input is received, domain, etc. A system may use the interaction affinity data to determine an alternative input representation for a spoken input to cause output of a desired response to the spoken input. The system may also use the interaction affinity data to recommend an action to a user.

Interaction data and processing natural language inputs

Techniques for determining an intent for a user input in a dialog are described. The system processes historic interaction data that is structured based skills and intents, with each skill-intent pair being associated with one or more past user inputs received by the system, one or more sample inputs, and one or more alternative representations of the user inputs. Based on processing of the historic interaction data and dialog data of previous turns of the dialog, the system determines potential intents for the user input of the current turn of the dialog. The potential intents may correspond to a presently active skill or another skill, enabling the user to interact with another skill during the dialog.

Dialog management system

Techniques for determining an intent of a subsequent user input in a dialog are described. The system processes historic interaction data that is structured based on natural language understanding (NLU) hypotheses, with each NLU hypothesis being associated with one or more past user inputs received by the system, one or more sample inputs, and one or more past system responses. Based on processing of the historic interaction data and dialog data of previous turns of the dialog, the system determines candidate intents for the subsequent turn of the dialog. The system also uses context data to determine the candidate intents.

Described herein is a system for responding to a frustrated user with a response determined based on spoken language understanding (SLU) processing of a user input. The system detects user frustration and responds to a repeated user input by confirming an action to be performed or presenting an alternative action, instead of performing the action responsive to the user input. The system also detects poor audio quality of the captured user input, and responds by requesting the user to repeat the user input. The system processes sentiment data and signal quality data to respond to user inputs.

Sentiment aware voice user interface

Techniques for handling errors during processing of natural language inputs are described. A system may process a natural language input to generate an ASR hypothesis or NLU hypothesis. The system may use more than one data searching technique (e.g., deep neural network searching, convolutional neural network searching, etc.) to generate an alternate ASR hypothesis or NLU hypothesis, depending on the type of hypothesis input for alternate hypothesis processing.

Alternate natural language input generation

Growing community of inventors

Redmond, WA, United States of America

Chenlei Guo