Amazon Technologies, Inc.

Melanie C B Gens

Xinhong Zhang

Rahul Gupta

Sachin Midha

Anoop Kumar

Sriram Venkatapathy

Pooja Sanjay Sonawane

Samuel Harry Ingbar

Pavel Bhowmik

Spyridon Matsoukas

Ying Shi

Angeliki Metallinou

Andrew Smith

Chengwei Su

Premkumar Natarajan

Wael Hamza

Jyoti Chhabra

Mark Lawrence

Md Humayun Arafat

Raja Bose

Arijit Biswas

Jwala Dhamala

Avni Khatri

Srinivasan Rengarajan Veeravanallur

Seyed Omid Razavi

Hann Wang

Matthew Cavell Johnson

Prashan Wanigasekara

Kechin Qin

Spurthideepika Sandiri

Apurv Verma

Qingwen Ye

Mayur Himmatbhai Dabhi

Zexiong Yan

Ryan McCrate

Ramya Chaganti

Dewan Muhammed Ibtesham

Jennifer Yuen

Techniques for performing an action with respect to displayed content are described. A natural language interpretation corresponding to a received spoken user input may be determined. Prior to receiving the spoken user input, content may be displayed to the user from which the spoken user input was received. The natural language interpretation may represent a request to perform an action with respect to a portion of the content currently being displayed. Content identifiers corresponding to content being displayed, may be determined, and embedding data representing at least one feature of the content may be determined using the content identifiers. The natural language interpretation and the embedding data may be processed to determine that the spoken user input relates to a first portion of the displayed content instead of a second portion of the displayed content. Based on the determination, an action responsive to the spoken user input may be performed.

Natural language interactions using visual understanding

Techniques for generating a prompt for a language model to determine an action responsive to a user input, are described. In some embodiments, the system receives a user input, determines one or more application programming interfaces (APIs) configured to perform actions that are relevant to the user input and exemplars representing examples of using the APIs with respect to user inputs similar to the current user input. The system further determines device states of devices that are determined to be related to the user input and also determines other contextual information (e.g., weather information, time of day, geographic location, etc.). The system generates a prompt including the user input, the APIs, the exemplars, the device states, and the other contextual information. A language model processes the prompt to determine an action responsive to the user input and the system causes performance of the action.

Natural language generation

Techniques for enabling access in a multi-assistant speech processing system are described, where a first assistant system may use components of a second assistant system as data processing components. Runtime operational data and user input data related to the first assistant may be kept separate from the processing data and input data related to the second assistant by propagating a first account ID, for user inputs directed to the first assistant, through the processing pipeline, and using a second account for user inputs directed to the second assistant. A mapping between the first account ID and the second account ID may be accessible to a select number of system components. Handoffs between the two assistants are handled in a manner where data related to one assistant is not accessible by the other assistant.

Data processing in a multi-assistant system

Techniques for evaluating a natural language understanding (NLU) component and determining an action to resolve an issue processing a user input are described. The system determines which component is invoked by a baseline NLU component is processing the user input, and which component is invoked by an updated NLU component. Based on that information, the system selects the action to resolve the updated NLU component generating an undesired response to the user input.

Evaluating natural language processing components

Devices and techniques are generally described for cohort determination in natural language processing. In various examples, a first natural language input to a natural language processing system may be determined. The first natural language input may be associated with a first account identifier. A first machine learning model may determine first data representing one or more words of the first natural language input. A second machine learning model may determine second data representing one or more acoustic characteristics of the first natural language input. Third data may be determined, the third data including a predicted performance for processing the first natural language input by the natural language processing system. The third data may be determined based on the first data representation and the second data representation.

Cohort determination in natural language processing

Techniques for tuning parameters for machine learning models are described. Different values for a parameter are tested to determine the value that results in an optimized model. A parameter value may be selected for testing using a search algorithm based on how the model performs with respect to other values for the parameter. Different values may be tested until a stopping criterion (such as time for testing, number of trials, amount of enhancement in performance, etc.) is met. In some embodiments, the techniques may be used to determine parameter values for natural language processing models.

Model configuration

Growing community of inventors

Seattle, WA, United States of America

Melanie C B Gens